TECHNICAL  REPORT  2001-003 


Single  Integrated  Air  Picture  (SIAP) 
Metrics  Implementation 


OCTOBER  2001 


SINGLE  INTEGRATED  AIR  PICTURE  (SIAP) 
System  Engineering 
Task  Force  (SE  TF) 

1931  Jefferson  Davis  Highway 
Crystal  Mall  3,  Suite  1 1 09 
Arlington,  VA  22203 


REPORT  DOCUMENTATION  PAGE 


Form  Approved  OMB  No. 
0704-0188 

Public  reporting  burder  for  this  collection  of  information  is  estibated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing 
and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burder  to  Department  of  Defense,  Washington 
Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of 
law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1 .  REPORT  DATE  (DD-MM- YYYY)  2.  REPORT  TYPE  3 .  DATES  COVERED  (EROM  -  TO) 

29-10-2001  Technical  Report  xx-xx-2001  to  xx-xx-2001 

4.  TITLE  AND  SUBTITLE 

Single  Integrated  Air  Picture  (SIAP)  Metrics  Implementation 

Unclassified 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Votruba,  Paul ; 

Nisley,  Rich  ; 

Rothrock,  Ron  ; 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PEREORMING  ORGANIZATION  NAME  AND  ADDRESS 

Single  Integrated  Air  Picture 

Attn:  CM3  Room  1 142 

1931  Jefferson  Davis  Hwy 

Arlington,  VA22202-3523 

8.  PEREORMING  ORGANIZATION  REPORT 
NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME  AND  ADDRESS 

JROC 

The  Joint  Staff 

Washington,  DC203 18-8000 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 

NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

APUBLIC  RELEASE 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

Metrics  have  been  identified  to  quantify  a  Single  Integrate  Air  Picture  (SIAP)  in  Technical  Reports  2001-001  and  2001-002.  Beyond  the  task 
of  defining  appropriate  measures  for  quantifying  the  SIAP,  however,  it  is  essential  to  establish  guidelines  for  implementation  to  ensure 
consistency  in  the  evaluation  and  use  of  the  measures.  In  particular,  objective  evaluation  of  the  quantitative  SIAP  attribute  measures,  crucial  to 
the  process  of  assessing  compliance  with  Theater  Air  and  Missile  Defense  (TAMD)  and  Combat  Identification  (CID)  Capstone  Requirements 
Documents  (CRDs),  is  dependent  upon  prior  establishment  of  a  track  data  base  to  measure  and  an  rule  for  determining  association  of  tracks 
with  truth  objects. 

15.  SUBJECT  TERMS 


Single  Integrated  Air  Picture;  SIAP;  Requirements;  CRD;  KPP;  Attributes;  Metrics;  TAMD;  CID;  Warfighting;  MOEs;  MOPs; 
Implementation;  assignment;  track  matching 


116.  SECURITY  CLASSIFICATION  OF: 


17.  LIMITATION 
OF  ABSTRACT 
Same  as  Report 
(SAR) _ 


18. 

NUMBER 
OF  PAGES 
66 


19.  NAME  OF  RESPONSIBLE  PERSON 

Fenster,  Lynn 

lfenster@dtic.mil 


a.  REPORT 


lb.  ABSTRACT 


|c.  THIS  PAGE 


|19b.  TELEPHONE  NUMBER 


Unclassified 


Unclassified 


Unclassified 


International  Area  Code 
Area  Code  Telephone  Number 


703767-9007 

DSN 

427-9007 


Standard  Form  298  (Rev.  8-98) 
Prescribed  by  ANSI  Std  Z39.18 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently 
valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1 .  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE  3.  DATES  COVERED  (From  -  To) 

29-10-2001  Technical  Report 


4.  TITLE  AND  SUBTITLE  5a.  CONTRACT  NUMBER 

Single  Integrated  Air  Picture  (SIAP)  Metrics  Implementation 

5b.  GRANT  NUMBER 


6.  AUTHOR(S) 

Votruba,  Paul;  Nisley,  Rich;  Rothrock,  Ron;  Authors 
Zombro,  Brett;  Editor 


5c.  PROGRAM  ELEMENT  NUMBER 

0603879N 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Single  Integrated  Air  Picture 
Attn:  CM3  Room  1142 

1931  Jefferson  Davis  Hwy 
Arlington  VA  22202-3523 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 

SIAP  SE  IF  TR  2001-003 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

JROC  USD(AT&L) 

The  Joint  Staff  3010  Defense  Pentagon 

Washington,  D.C.  20318-8000  Washington,  DC  20301-3010 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

A  Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

Updated  annually. 


14.  ABSTRACT 

Metrics  have  been  identified  to  quantify  a  Single  Integrate  Air  Picture  (SIAP)  in  Technical 
Reports  2001-001  and  2001-002.  Beyond  the  task  of  defining  appropriate  measures  for 
quantifying  the  SIAP,  however,  it  is  essential  to  establish  guidelines  for  implementation  to 
ensure  consistency  in  the  evaluation  and  use  of  the  measures.  In  particular,  objective 
evaluation  of  the  quantitative  SIAP  attribute  measures,  crucial  to  the  process  of  assessing 
compliance  with  Theater  Air  and  Missile  Defense  (TAMD)  and  Combat  Identification  (CID) 
Capstone  Requirements  Documents  (CRDs),  is  dependent  upon  prior  establishment  of  a  track  data 
base  to  measure  and  an  rule  for  determining  association  of  tracks  with  truth  objects. 


15.  SUBJECT  TERMS 

Single  Integrated  Air  Picture;  SIAP;  Requirements;  CRD;  KPP;  Attributes;  Metrics;  TAMD;  CID; 
Warfighting;  MOEs;  MOPs;  Implementation;  assignment;  track  matching 


19a.  NAME  OF  RESPONSIBLE  PERSON 

LCDR  Paul  M.  Votruba 


19b.  TELEPHONE  NUMBER  (mclude  area 
code) 

703-602-6441 


16.  SECURITY  CLASSIFICATION  OF: 

a.  REPORT 

UNCLASSIFIED 

b.  ABSTRACT 

UNCLASSIFIED 

C.  THIS  PAGE 

UNCLASSIFIED 

17.  LIMITATION 

18.  NUMBER 

OF  ABSTRACT 

OF  PAGES 

Same  As 

64 

Report 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39.18 


TECHNICAL  REPORT  2001-003 


RADM  Mike  Mathis 


Single  Integrated  Air  Picture  (SIAP) 
Metrics  Implementation 


OCTOBER  2001 


SINGLE  INTEGRATED  AIR  PICTURE  (SIAP) 
System  Engineering 
Task  Force  (SE  TF) 

1931  Jefferson  Davis  Highway 
Crystal  Mall  3,  Suite  1109 
Arlington,  VA  22203 


Approved  for  public  release: 
distribution  is  unlimited 


Page  2 
Version  1 .0 
October  25,  2001 


GOVERNMENT  RIGHTS  IN  DATA  STATEMENT 


Reproduction  of  this  publication  in  whole 
or  in  part  is  permitted  for  any  purpose 
of  the  United  States  Government. 


Page  3 
Version  1 .0 
October  25,  2001 


FOREWORD 


List  of  Contributors 

The  SIAP  Metrics  Implementation  Technical  Report  is  the  result  of  collective 
efforts  of  members  of  the  SIAP  Attributes  and  Metrics  Working  Group,  who  drafted  the 
content  of  the  report  through  several  face-to-face  meetings,  teleconferences,  and 
electronic  mail  exchanges  spanning  the  period  from  July  to  October,  2001 .  The 
following  individuals  contributed  to  the  report  through  their  participation  in  either  live  or 
virtual  meetings  of  the  Working  Group: 

LCDR  Paul  Votruba,  SIAP  SE  TF  (Chair) 

Jim  Africa,  USN/NSWC  Dahlgren 

Tom  Hart,  USMC/Mitre 

John  Jordan,  USA/CAS 

Larry  Lewis,  JCIET/CNA 

Jeff  Lutz,  JCIET/Mevatec 

Jeff  Nicoll,  SIAP  SETF/IDA 

Rich  Nisley,  JTAMDO/SAIC 

John  Nordmann,  USAF/Mitre 

Preston  Poore,  USN/PCI 

Jeff  Renegar,  USN/NSWC  Dahlgren 

Ron  Rothrock,  SIAP  SE  TF/BMDO/Sparta 

Paul  Symborski,  JCIET/CNA 

Erik  Van  Fleet,  USN/NSWC  Corona 

Ray  Washburn,  USA/PEO  AMD 

Brett  Zombro,  SIAP  SE  TF/SPA 


Page  4 
Version  1 .0 
October  25,  2001 


List  of  Contributors  (continued) 


The  service  and  agency  review  process  brought  some  additional  input,  revision, 
and  commentary  to  bear  on  the  report,  and  through  this  process  the  contributions  of  a 
number  of  other  individuals  and  organizations  are  reflected  in  the  final  document.  In 
addition  to  names  already  noted,  contributors  to  the  review  process  include; 

Jeff  Allison,  USN 

Maj  Mark  Arbogast,  USAF/USD(AT&L) 

Carla  Barrett,  JCIET/CNA 

Jon  Barto,  SIAP  SE  TF/Mitre 

Dan  Bergstrom,  USN/NSWC  Corona 

Alex  Brofos,  USMC/Mitre 

Joan  Cartier,  SIAP  SE  TF/IDA 

Jack  Connor,  JTAMDO/SPA 

Gary  Demas,  ASD  C3I 

Danny  Ellenburg,  USA/Intergraph 

John  Flynn,  BMDO 

Barry  Fridling,  JTAMDO 

Alan  Glazman,  USN/NSWC  Dahlgren 

John  Hamilton,  OSD-DOT&E  R&R 

Kent  Haspert,  JTAMDO/IDA 

Chris  Jones,  JTAMDO/CSCI 

Grant  Klokeid,  USN/NSWC  Corona 

LCDR  Lassell,  USN/PMS  465 

Ernie  MacNamara,  CNA 

Maj  John  Mades,  USMC/MARCORSYSCOM 

Alvin  Murphy,  USN/NSWC  Dahlgren 

Ralph  O'Connell,  OSD-DOT&E  R&R 

Robert  O’Donohue,  OSD 

Steve  Pick,  JTAMDO 

Crystle  Pishon,  USAF/Mitre 

James  Rebel,  RDA  CHENG 

LCDR  Tim  Spitser,  SIAP  SE  TF 

Peter  Stafford,  USN/NSWC-Dahlgren 

LTC  Lloyd  Stephenson,  JTAMDO 

Howard  Sterling,  OSD-ATL 


Page  5 
Version  1 .0 
October  25,  2001 


EXECUTIVE  SUMMARY 


PROBLEM 

The  SIAP  Implementation  Plan  states  that  the  overarching  objective  of  the  SIAP  SE  TF 
is  "to  identify  incremental  improvements  in  the  ...  SIAP  capability  that  will  provide 
commensurate  incremental  improvements  in  warfighter  capabilities."  Metrics  have  been 
identified  to  quantify  a  SIAP  and  these  incremental  improvements  (SIAP  SE  TF 
Technical  Reports  2001-001  and  2001  -002).  Beyond  the  task  of  defining  appropriate 
measures  for  quantifying  the  SIAP,  however,  it  is  essential  to  establish  guidelines  for 
implementation  to  ensure  consistency  in  the  evaluation  and  use  of  the  measures.  In 
particular,  objective  evaluation  of  the  quantitative  SIAP  attribute  measures,  crucial  to  the 
process  of  assessing  compliance  with  Theater  Air  and  Missile  Defense  (TAMD)  and 
Combat  Identification  (CID)  Capstone  Requirements  Documents  (CRDs),  is  dependent 
upon  prior  establishment  of  a  track  data  base  to  measure  and  an  rule  for  determining 
association  of  tracks  with  truth  objects.  The  mathematical  definitions  of  the  SIAP 
attributes  cannot  be  applied  consistently  without  an  understanding  of  the  measurable 
data  base  and  the  criteria  for  tracks-to-truth  "assignment."  Decisions  must  be  made  as 
to  how  rigorously  the  assignment  criteria  are  to  be  specified,  how  broadly  they  are  to  be 
applied,  and  how  and  under  what  conditions  they  should  be  modified.  Some  of  these 
decisions  involve  considerations  of  possible  computational  approaches.  There  are 
related  empirical  issues,  many  specific  to  particular  test  environments,  that  may  need  to 
be  addressed.  For  example,  a  live  experiment  may  yield  insufficient  data  to  carry  out  an 
agreed-upon  assignment  rule.  Some  of  these  issues  are  presently  known;  others  may 
arise  as  the  SIAP  assessment  process  evolves.  SIAP  metrics  implementation  is  the 
term  used  in  this  report  to  encompass  all  of  these  assessment  issues. 

OBJECTIVES 

This  technical  report  has  the  following  objectives. 

1)  Establish  consensus  on  which  tracks  will  be  included  in  defining  a  SIAP  (the 
measurability  problem). 

There  are  many  track  types  that  can  be  considered  for  scoring  the  SIAP  metrics.  It  is 
paramount  that  there  be  consensus  on  which  ones  will  be  counted  to  ensure  proper 
comparisons,  and  that  the  classes  of  tracks  included  (as  well  as  excluded)  be  precisely 
defined. 

2)  Establish  a  tracks-to-truth  assignment  procedure  applicable  across  a  variety  of 
testing  environments,  and  address  technical  issues  of  algorithmic  implementation 
(the  assignment  problem). 

Justification  of  choices  made  and  recommendations  for  later  refinements  are  to  be 
included.  Technical  issues  include  casting  the  procedure  in  a  mathematically  explicit 
form,  and  addressing  data  interpolation. 

3)  Define  and  address  issues  pertaining  to  live  test  evaluations  in  connection  with 
the  proposed  measurability  and  assignment  approaches  (reconstruction 
problems). 

As  already  suggested,  the  presently  identified  issues  will  be  discussed  here,  but  more 
issues  are  certain  to  arise  as  testing  and  evaluation  gets  underway.  Thus,  this  objective 
can  only  be  met  in  a  provisional  manner  in  this  report. 
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4)  Provide  alternative  track  bases  and  excursions  to  scoring  methods  to  allow 
comparison  with  the  proposed  approaches. 

SIAP  system  engineering  will  face  many  assessment  problems  beyond  the  realm  of 
current  assessments,  on  which  the  measurability  and  assignment  studies  are  focused. 

A  framework  should  be  provided  for  extending  measurability  to  other  track  data  bases, 
and  for  implementing  alternative  assignment  and  scoring  procedures  should  the  current 
proposals  require  reevaluation. 

APPROACH 

The  approach  to  measurability  is  to  adopt  standardized  or  commonly  accepted 
definitions  of  the  track  data  bases  to  be  considered,  and  to  specifically  identify  those 
which  constitute  the  information  which  an  operator  or  commander  can  meaningfully  act 
upon  in  the  context  of  legacy  SIAP-related  systems.  Link-1 6  is  used  as  an  initial  context 
for  most  of  the  formal  definitions,  but  the  measurability  recommendations  are  made  with 
broader  applicability  in  mind.  Discussion  of  alternatives  and  justification  of  choices 
made  are  provided.  The  approach  to  assignment  is  to  define  a  deterministic 
(algorithmic)  assignment  procedure  which  covers  assignment  concerns  in  both  live 
evaluation  and  constructive  simulation  contexts.  The  assignment  procedure  is  based 
on  an  optimization  method  (minimization  of  an  appropriate  assignment  cost  function), 
but  the  optimization  criterion  is  combined  with  gating  and  nearest-neighbor  multiple- 
track  assignment  possibilities  in  a  way  designed  to  capture  the  benefits  of  all  of  these 
approaches.  Benefits  and  disadvantages  are  discussed.  The  cost  function  and  related 
algorithmic  issues  are  addressed  in  a  mathematically  explicit  manner.  The  approach  to 
reconstruction  is  to  discuss  the  known  issues  and  make  recommendations  for  their 
resolution  to  the  extent  possible.  Special  cases  such  as  formation  tracking,  time 
alignment,  and  test  design  issues  are  addressed  and  tentative  recommendations  are 
provided  for  each.  Finally,  alternatives  to  the  proposed  measurability  and  assignment 
approaches,  providing  a  starting  point  for  extending  the  results  of  this  report  to  meet 
future  assessment  needs,  are  discussed. 

FINDINGS 

The  track  scoring  approach  presented  identifies  those  measurable  tracks  that  should  be 
used  in  evaluating  SIAP  metrics.  There  is  a  brief  discussion  on  how  exceptions  to  the 
proposed  scoring  policy  may  be  in  order  for  ballistic  missile  tracks.  A  detailed 
assignment  algorithm  and  its  associated  cost  function  are  also  presented  along  with 
examples  and  rationale  for  their  implementation.  Both  proposals  can  be  adapted  to 
simulation  as  well  as  live  exercise  data  reconstruction  for  defining  a  SIAP  through  the 
SIAP  attributes.  An  appendix  includes  mathematical  details  on  the  theory  by  which  the 
assignment  cost  function  (for  optimal  assignment)  was  derived.  Another  appendix 
provides  mathematical  details  on  a  truth  data  interpolation  procedure  for  use  in  the 
assignment  algorithm. 

CONCLUSIONS 

The  recommended  approaches  to  assignment  and  reconstruction  provide  a  consistent 
preliminary  basis  upon  which  to  evaluate  SIAP  metrics  and  quantify  SIAP  capabilities. 
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RECOMMENDATIONS 

Institutionalize  the  approach  defined  herein  as  to  the  methodology  for  performing  SIAP- 
related  evaluation  on  all  units  where  SIAP  metrics  are  calculated  and  evaluate  approach 
for  applicability  to  other  missions  and  domains.  Explore  the  excursions  and  assignment 
alternatives  identified  herein  for  their  potential  applicability  to  future  refinement  and 
reevaluation  of  the  methodology. 
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1.  INTRODUCTION 


A  methodology  for  evaluating  the  Single  Integrated  Air  Picture  (SIAP)  by  way  of  a 
hierarchy  of  well-defined  SIAP  metrics  has  been  established  in  two  related  technical 
reports  (SIAP  SE  TF  Technical  Reports  2001  -001  and  2001  -002).  This  methodology  is 
intended  to  measure  the  degree  to  which  the  ability  to  build  and  maintain  a  SIAP  is 
achieved  -  an  objective  which  will  in  turn  contribute  to  the  assessment  and  forecasting 
of  progress  over  time.  In  addition,  the  process  of  evaluating  capability  in  this  way  gives 
significant  insight  into  specific  issues  that  must  be  resolved. 

The  SIAP  assessment  approach  must  be  standardized  to  ensure  uniformity  and 
objectivity  in  the  evaluation  of  capabilities.  Ultimately,  the  ability  to  make  good  system 
engineering  decisions  towards  the  achievement  of  a  SIAP  capability  will  crucially 
depend  upon  this  ability  to  make  objective  comparisons  and  assessments.  This 
technical  report  is  devoted  to  the  standardization  of  the  SIAP  assessment  methodology 
through  the  establishment  of  a  self-correcting  path  towards  uniform  and  objective 
evaluations  of  SIAP  metrics. 

A  number  of  practical  issues  arise  when  one  attempts  to  give  objective  values  to 
SIAP  metrics  on  the  basis  of  data  from  live  tests  or  high-fidelity  simulations.  The  raw 
data  collected  generally  does  not  yield  the  desired  metric  directly.  "Implementation"  of  a 
SIAP  metric  (or  class  of  metrics)  encompasses  all  such  issues  of  practical 
measurement,  apart  from  the  strict  definition  of  the  metric(s).  The  concern  is  primarily 
with  the  SIAP  attributes,  since  these  are  linked  to  Theater  Air  and  Missile  Defense 
(TAMD)  and  Combat  Identification  (CID)  Capstone  Requirements  Documents  (CRDs) 
(2001 ),  although  many  of  the  same  issues  will  arise  with  metrics  at  other  levels  in  the 
SIAP  metrics  hierarchy  (cf.  SIAP  SE  TF  Technical  Report  2001  -002). 

Issues  identified  so  far  have  been  grouped  into  the  following  three  areas; 
measurability,  assignment,  and  other  reconstruction  issues.  The  measurability  issue 
concerns  what  track  data  is  to  be  used  for  SIAP  assessments.  Track  assignment 
concerns  the  procedure  to  be  followed  for  determining  which  tracks  are  “assigned”  to 
which  truth  objects;  specifically,  how  and  to  what  degree  this  procedure  should  be 
automated.  Reconstruction  comprises  track  assignment  and  other  issues,  mostly 
pertaining  to  live  exercise  evaluations,  which  involve  the  construction  of  track-to-object 
assignments  from  experimental  data.  Many  of  these  issues  are  concerned  with  making 
allowance  in  test  analysis  for  difficulties  in  obtaining  complete  track  data  or  interpreting 
the  track  data  that  is  obtained.  It  is  the  objective  of  this  report  to  describe  each  of  these 
implementation  issues  in  detail,  and  make  specific  proposals  and  recommendations 
towards  their  resolution. 
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2.  MEASURABILITY 


The  measurability  problem  refers  to  the  problem  of  specifying  a  well-defined  set 
of  tracks  on  which  to  base  SIAP  assessments.  The  formal  definition  of  a  SIAP-  the 
“product  of  fused,  common,  continual,  unambiguous  tracks  of  airborne  objects  in  the 
surveillance  area”  (Theater  Air  and  Missile  Defense  Capstone  Requirements  Document 
(TAMD  CRD))  -  does  not  define  what  tracks  are  to  be  considered.  This  is  deliberate, 
allowing  for  evolution  of  the  concept  of  a  SIAP  as  network  technology  evolves. 

There  is  understandably  more  interest  in  attaining  a  SIAP  on  that  part  of  the 
aerospace  picture  which  can  be  meaningfully  acted  upon  -  for  example,  tracks  used  for 
the  purpose  of  allocating  weapons,  or  at  least  for  making  a  decision  to  fire  (battle 
management).  What  this  means  for  current  assessments,  though,  may  be  quite 
different  for  what  it  may  mean  for  long-term  planning  and  forecasting.  Even  for  current 
assessments,  delineating  precisely  those  tracks  that  are  “actionable”  when  faced  with 
multiple  networks  operating  simultaneously  on  different  principles  involves  some 
complications. 

The  approach  of  this  report  focuses  on  current  assessments.  Other  classes  of 
tracks  identified  outside  this  chapter  will  be  considered  “excursions”  in  Section  5  to  be 
investigated  for  potential  application  to  other  aspects  of  SIAP  system  engineering 
analysis.  The  initial  approach  to  measurability  described  in  the  following  two  sections 
has  been  guided  by  considerations  of  issues  pertaining  to  tactical  data  links  such  as 
Link-1 6,  but  is  expected  to  be  applicable  in  other  contexts  in  all  essential  respects.  As 
the  future  analytical  focus  shifts  to  other  kinds  of  networks,  some  refinement  of 
terminology  may  be  needed  in  order  to  maintain  the  intended  emphasis  on  actionable 
information. 

2.1  Definition  of  Terms  and  Assumptions 

The  following  definitions  of  different  track  types  are  relevant  to  the  discussion  of 
track  measurability  issues. 

Track  -  "(1 )  the  graphic  and/or  alphanumeric  representation  of  an  object, 
point,  or  bearing  whose  position  and/or  characteristics  are  collated  from 
sensors  and/or  other  data  sources;  (2)  a  collated  set  of  data  associated 
with  a  track  number  for  the  purpose  of  representing  the  position  and/or 
characteristics  of  a  specific  object,  point,  or  bearing"  (MIL-STD-601 6A, 

1 999).  For  SIAP  assessment  purposes,  a  track  is  understood  as  an 
actionable  track,  not  to  include  tentative  tracks  or  clutter  tracks. 

Local  Track  - "  A  track  established  within  an  interface  unit  based  on 
locally  entered  positional  information.  Amplifying  data  associated  with  the 
track  may  be  derived  locally,  from  supporting  units,  or  from  data  links" 

(MIL-STD-601 6A,  1999). 
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For  the  purposes  of  metrics  implementation  "positional  information"  is  understood  to 
consist  of  any  subset  of  angle,  range,  and  range-rate  measurement  components  and 
their  error  covariance,  as  well  as  filtered  kinematic  state  estimates  and  their  error 
covariance.  The  term  "interface  unit"  refers  specifically  to  participants  on  a  tactical 
digital  information  link  (TADIL),  but  these  definitions  are  understood  to  be  generally 
applicable  to  any  participant  contributing  to  or  using  the  SIAP.  Local  tracks  include 
single-sensor  tracks  generated  with  local  sensor  data  and  single-platform  multi-sensor 
composite  tracks  generated  with  data  from  multiple  local  sensors.  Local  tracks  also 
include  multi-platform,  multi-sensor  composite  tracks,  such  as  tracks  generated  by  the 
Cooperative  Engagement  Capability  (CEC)  or  “Joint  Composite  Tracking  Network” 
(“JCTN”),  provided  the  composite  tracks  are  locally  filtered/processed,  i.e.;  the 
positional  information  is  locally  entered  (which  need  not  always  be  the  case  when  CEC 
or  “JCTN”  tracks  are  passed  over  Link-1 6,  say). 

Remote  Track  -  "A  track  established  within  an  interface  unit  based  upon 
positional  information  derived  from  a  data  link  report  or  reports. 

Amplifying  data  associated  with  the  track  may  be  derived  locally,  from 
supporting  units,  or  from  data  links"  (MIL -STD -601 6A,  1 999). 

Local-Only  Track  -  A  local  track  that  is  not  correlated  to  a  remote  track  at 
the  time  of  evaluation. 

Local-Mutual  Track  -  A  local  track  that  is  correlated  to  a  remote  track  at 
the  time  of  evaluation. 

Remote-Only  Track  -  A  remote  track  that  is  not  correlated  to  a  local  track 
at  the  time  of  evaluation. 

Remote-Mutual  Track  -  A  remote  track  that  is  correlated  to  a  local  track  at 
the  time  of  evaluation. 

Pending  Track  -  "A  track  which  has  not  been  subjected  to  the 
identification  process"  (MIL-STD-6016A,  1999). 

Compliance  of  systems  with  SIAP-related  requirements  will  be  assessed  on  the 
basis  of  SIAP  attributes.  SIAP  attributes  are  based  on  the  actionable  track  data  in  the 
central  track  stores  (CTS)  of  participants,  where  GTS  is  defined  by  the  particular  type  of 
system.  The  CTS  contains  all  of  the  track  types  defined  in  this  section.  In  accordance 
with  the  definition  of  "track"  cited  above,  a  track  is  considered  actionable  once  it  has  an 
associated  track  number  in  the  form  of  a  central  track  store  locator  (CTSL).  Since  this 
qualification  implies  nothing  about  the  status  of  the  identification  process,  actionable 
tracks  include  pending  tracks  as  defined  above.  Pending  tracks  are,  at  least  in  some 
cases,  "available  for  reporting"  (MIL-STD-601 1 B,  1999).  While  many  legacy  systems 
are  equipped  with  filters  which  prevent  local  pending  tracks  from  being  reported  on  Link- 
1 6,  this  is  not  a  universal  feature  of  legacy  systems,  and,  even  if  pending  tracks  are  not 
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reported,  as  local  tracks  they  are  displayable  and  may  influence  an  operator's  decisions. 
Local  pending  data  may  also  be  subject  to  correlation  attempts.  Pending  tracks,  once 
they  have  acquired  CTSLs,  therefore  count  as  actionable  tracks  for  the  purposes  of  this 
report  and  for  all  SIAP  attribute  evaluations. 

However,  mutual  tracks,  as  well  as  certain  other  classes  of  correlated  tracks, 
raise  special  issues,  which  require  further  restricting  the  class  of  measurable  tracks. 

2.2  Scoring  Preferences 

There  are  a  number  of  issues  remaining  on  the  scoring  of  tracks.  The  first 
involves  remote-mutual  tracks.  To  avoid  double-counting  correlated  tracks  that  are 
understood  by  a  participant  as  containing  two  versions  of  the  same  information,  it  will  be 
assumed  for  current  assessments  that  one  shall  always  use  the  local-mutual  tracks. 

The  scoring  of  local-mutual  rather  than  remote-mutual  tracks  indicates  preference  for 
what  are  usually  regarded  as  the  more  "actionable”  tracks  for  most  legacy  air  and 
missile  defense  systems.  Furthermore,  local  tracks  are  always  available  in  all  legacy 
systems,  whereas  remote  mutual  track  data  is  not  always  retained  after  the  correlation 
process  of  some  SIAP  systems.^ 

The  second  issue  addresses  multiple  local  tracks  from  independent  sources. 

This  is  not  an  issue  for  AEGIS  and  PATRIOT  systems  because  these  systems 
automatically  filter  tracks  not  meeting  certain  criteria  before  they  enter  the  GTS.  Not  all 
SIAP  systems  have  an  automatic  filtering  capability;  therefore,  for  these  systems,  a 
scoring  precedence  must  be  defined. 

Other  issues  include  local-to-local  correlations,  remote-to-remote,  as  well  as 
network-to-network  correlations.  Again,  these  issues  are  system-dependent.  Scoring 
preferences  should  ideally  be  based  on  models  of  the  internal  precedence  logic  for  each 
type  of  correlation,  but  such  models  may  be  difficult  to  come  by. 

The  range  of  possibilities  makes  it  unlikely  that  a  single  precedence  rule  will  be 
adequate  for  all  purposes,  but  this  issue  is  still  under  discussion.  For  present  purposes, 
it  is  important  that  the  precedence  be  clearly  defined  at  least  on  a  case-by-case  basis 
whenever  the  SIAP  attributes  are  applied  to  scoring  of  test  or  simulation  results. 

Table  1  gives  an  indication  of  the  types  of  tracks  that  can  be  measured  and  the 
associated  track  data  for  each.  In  summary,  the  SIAP  SE  TF  will  initially  base  SIAP 
attribute  and  applicable  MOP  calculations  on  track  data  in  participants'  central  track 
stores,  excluding  remote-mutual  tracks.  The  information  that  will  be  used  is  marked  in 
red  (the  bold  ‘X’). 


^  The  assertions  in  the  text  refer  to  iegacy  U.S.  systems.  There  are,  however,  exceptions  invoiving 
certain  iegacy  U.K.  systems,  some  of  which  are  to  be  inciuded  in  iive  exercises  of  interest  for  near-term 
SIAP  assessments  (in  particuiar,  JCIET  02).  For  exampie,  there  are  systems  on  some  U.K.  frigates 
which  process  and  dispiay  remote-mutuai  tracks  in  preference  to  iocai-mutuais.  For  these  exceptionai 
systems,  the  remote-mutuais  couid  be  considered  the  more  actionabie  tracks,  and  it  wouid  be  consistent 
with  the  phiiosophy  of  this  report  to  reverse  the  usuai  scoring  precedence  ruie  in  just  these  cases. 
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Local 

Remote 

Mutual 

System  track  number 

X 

X 

X 

Link  track  number 

X 

X 

X 

Local  position/velocity 

X 

X 

Remote  position/velocity 

X 

X 

X  -  Track  information  available 
X  -  T rack  information  to  be  used  for  SIAP  assessment 


Table  1 .  Tracks  to  measure. 


3.  ASSIGNMENT 

The  computation  of  all  SIAP  attributes,  as  well  as  that  of  certain  track-based 
SIAP  MOPs,  is  dependent  upon  a  prior  knowledge  of  how  tracks  are  assigned  to 
objects,  (cf.  SIAP  SE  TF  Technical  Reports  2001-001  and  2001  -002,  and  Appendix  A 
of  this  report).  Provided  that  such  an  assignment  is  specified,  these  track-based 
metrics  are  uniquely  defined  quantities,  derivable  from  measurable  characteristics. 
However,  changing  the  assignment  will  generally  change  the  outcome  of  the 
calculations,  even  if  the  track  and  object  data  sets  used  are  unaltered.  The 
specification  of  how  the  assignment  should  be  done  can  thus  be  considered  a  major 
component  of  the  task  of  providing  clearly  defined  procedures  for  the  computation  of 
SIAP  metrics.  This  section  addresses  assignment  as  an  issue  independent  of  other 
implementation  questions.  However,  as  with  the  measurability  problem,  the  primary 
objective  is  to  obtain  a  viable  methodology  for  current  and  near-term  SIAP 
assessments,  for  which  some  use  of  simplifying  assumptions  is  probably  unavoidable. 
Problems  arising  with  the  proposed  approach  will  have  to  be  dealt  with  on  a  case-by- 
case  basis  until  initial  assessment  experience  is  sufficient  to  indicate  more  attractive 
options. 

The  assignment  (or  lack  thereof,  in  the  case  of  spurious  tracks)  of  each  track  to  a 
specified  object  will  be  referred  to  in  the  remainder  of  this  report  as  a  "tracks-to-truth" 
assignment,  in  keeping  with  the  common  use  of  that  phrase  in  the  testing  and 
simulation  communities.  Recall  that  the  term  "object"  as  formally  defined  in  SIAP  SE  TF 
Technical  Report  2001  -001 ,  is  only  used  in  reference  to  what  are  commonly  called 
"truth"  objects;  thus  the  use  of  the  term  "truth"  does  not  imply  any  additional 
qualification. 

3.1  Definition  of  Terms  and  Assumptions 

In  the  following  treatment  of  the  tracks-to-truth  assignment  procedure,  it  is 
assumed  that  a  suitable  set  of  measurable  tracks  has  been  identified  as  per  Section  2, 
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and  in  particular  that  a  set  of  tracks  with  scoring  precedence  has  been  identified  from 
among  the  candidate  tracks  held  in  each  participant's  GTS.  It  is  also  assumed  that  a 
complete  set  of  "truth"  data  (i.e.,  position,  velocity,  force  membership,  and  other 
relevant  characteristics  for  each  object)  is  available  as  needed  at  every  time  throughout 
the  evaluation  period.  These  data  may  be  fixed  by  prior  determination  (in  some 
modeling  environments),  or  obtainable  through  direct  measurement  or  reliable 
interpolation  between  measurements.  The  assumption  that  truth  is  always  available  is  a 
precondition  that  may  be  difficult  to  meet  in  the  case  of  live  test  data  (cf.  Section  4.1  of 
this  report),  and  it  may  be  necessary  to  tailor  evaluation  intervals  after  the  fact  to  ensure 
that  it  is  satisfied.  This  assumption  should  not  present  any  difficulty  for  simulations 
however. 

The  following  specialized  terminology  is  useful  in  discussing  various  approaches 
to  assignment. 

Assignment  Function  -  The  formalized  end  result  of  a  tracks-to -truth 
assignment.  For  each  participant  m  and  each  scoring  time  t,  the 
assignment  function  maps  each  track  (from  among  those  with  scoring 
preference)  held  by  m  at  time  t,  either  to  a  particular  object,  or  to  no  object 
(the  latter  case  indicating  that  the  track  is  spurious). 

Since  the  assignment  function  generally  represents  a  discrete  data  set,  an  explicit 
mathematical  representation  of  the  function  is  not  essential  for  practical  computations  - 
the  track-object  associations  may  be  represented  by  an  array  or  any  other  convenient 
data  structure. 

Assignment  Procedure  -  Any  process  (whether  explicitly  formalized  or 
not)  by  which  an  assignment  function  is  derived,  at  least  partly  on  the 
basis  of  track  data  and  truth. 

Assignment  Algorithm  -  An  explicit,  deterministic  assignment  procedure 
which  relies  exclusively  on  track  data  and  truth. 

An  assignment  algorithm  is  thus  a  special  case  of  an  assignment  procedure  which  may 
be  regarded  as  a  suitable  candidate  for  automation  (through  software  coding).  More 
generally,  assignment  procedures  may  involve  elements  of  human  decision-making 
(which  are  not  easily  encoded),  may  rely  on  information  not  strictly  derivable  from  track 
data  (for  example,  anecdotal  information  supplied  by  human  participants,  also  not  easily 
encoded),  and  may  invoke  probabilistic  criteria. 

Gated  Assignment  -  An  assignment  procedure  which  automatically 
prohibits  any  instantaneous  track-to-object  assignment  if  the  track  does 
not  satisfy  a  pre-specified  condition  of  proximity  to  the  truth  object  ("gating 
constraint")  in  position,  and  possibly  also  in  velocity  or  other  amplifying 
data.  If  truth  data  is  known  for  all  (reportable)  objects,  then  a  track 
prohibited  from  all  such  assignments  by  gating  constraints  is  regarded  as 
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spurious,  and  an  object  similarly  prohibited  is  regarded  as  untracked.  (As 
presented  below,  an  object  may  be  regarded  as  untracked  for  additional 
reasons.)  Gating  criteria  are  re-evaluated  at  each  instant  in  time  for  which 
the  assignment  procedure  is  carried  out. 

Unique  Assignment  (UA)  -  An  assignment  procedure  which  results  in  a 
one-to-one  assignment  between  a  subset  of  eligible  tracks  and  a  subset  of 
eligible  objects,  where  "eligibility"  may  be  based  on  gating  or  other  criteria. 

In  other  words,  each  track  which  is  assigned  to  an  object  is  assigned  to  a 
unique  object,  but  tracks  may  be  unassigned  by  failing  to  meet  eligibility 
criteria,  or  because  there  are  more  tracks  than  objects.  If  there  are  fewer 
eligible  tracks  than  eligible  objects,  then  the  assignment  will  cover  no 
more  objects  than  there  are  eligible  tracks,  and  the  remaining  objects 
(formally,  those  outside  the  range  of  the  assignment  function)  are 
regarded  as  untracked. 

Unique  Optimal  Assignment  (UOA)  -  A  unique  assignment  procedure  in 
which  the  assignment  function  is  determined  as  that  which  optimizes  (by 
convention,  minimizes)  the  instantaneous  value  of  a  pre-specified 
assignment  cost  function  over  the  set  of  all  assignments  satisfying  UA 
eligibility  criteria.  The  cost  function  may  depend  only  on  the  instantaneous 
assignment,  or  may  depend  as  well  on  assignments  already  made  over 
some  previous  time  interval  (UOA  with  hysteresis).  A  gating  constraint 
may  also  be  imposed,  resulting  in  a  gated  unique  optimal  assignment 
(GUOA). 

Typically,  the  cost  function  specified  for  a  UOA  procedure  is  a  weighted  sum  of  all 
kinematic  state  estimate  errors  encompassed  by  a  chosen  assignment,  possibly 
modified  to  include  hysteresis.  A  UOA  procedure  with  such  a  cost  function  is  thus 
roughly  a  global  error  minimization  procedure. 

Independent  Nearest  Neighbor  Assignment  (INNA)  -  An  assignment 
procedure  which  instantaneously  assigns  every  track  to  the  nearest 
object,  possibly  subject  to  gating  constraints,  but  generally  without  regard 
for  uniqueness  or  for  how  other  tracks  may  be  assigned. 

Any  procedure  which  leads  to  a  definite  assignment  function  can  be  used  as  a 
basis  for  evaluation  of  SIAP  attributes  (and  other  measures).  Once  the  assignment 
function  is  determined,  the  extraction  of  a  small  (and  definite)  number  of  additional 
critical  variables  from  the  track  and  truth  data  completes  the  set  of  information  required 
for  the  explicit  computation  of  all  SIAP  attributes.  Appendix  A  lists  these  additional 
variables,  and  demonstrates  their  sufficiency  (taken  together  with  the  assignment 
function)  for  the  evaluation  of  the  SIAP  attributes. 
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3.2  Proposed  Approach  to  Assignment:  Examples  and  Rationale 


Briefly  stated,  the  SIAP  SE  TF  will  implement  a  multi-step  assignment  algorithm 
for  current  SIAP  assessments,  incorporating  the  following  features: 

(1)  assignment  limited  to  tracks  specified  as  having  scoring  precedence 

(2)  administrative  assignment  of  remote-only  self  tracks,  such  as  Link  1 6  J2.2  Air 
Precise  Participant  Location  and  Identification  (PPLI)  tracks 

(3)  a  position/velocity  and  other  set  of  gating  constraints  to  rule  out  implausible 
assignments  and  identify  spurious  tracks 

(4)  for  each  track  source  independently,  a  first  pass  through  a  GUOA  algorithm  with 
all  tracks  and  objects  not  excluded  by  gating  considered  eligible 

(5)  for  each  track  source  independently,  a  second  pass  through  a  gated  INNA 
algorithm  (with  the  same  gating  criteria  as  used  in  the  first  pass)  for  all  tracks  not 
assigned  in  the  first  pass 

The  proposed  procedure  is  outlined  in  greater  detail  in  Section  (3.3).  In  the 
remainder  of  this  section,  the  rationale  underlying  some  of  the  elements  of  the 
procedure  is  discussed. 

3.2.1  The  Use  of  an  Assignment  Algorithm 

First,  the  proposal  to  use  an  automated  assignment  algorithm,  as  opposed  to  a 
less  formal  assignment  procedure,  requires  some  discussion.  Analysis  of  military  field 
test  data  has  typically  relied  on  combinations  of  formal  and  informal  track  assignment 
procedures,  with  a  strong  contribution  from  the  data  analysts'  experience  and  common 
sense  judgements.  These  procedures  have  evolved  over  time  and  have  proven  to  be 
useful.  However,  they  are  time  and  labor  intensive. 

On  the  other  hand,  Monte  Carlo  simulations  designed  to  collect  statistical 
distributions  of  track  data  from  thousands  of  runs  essentially  demand  the  use  of 
automated  assignment  algorithms.  Assignment  procedures  which  involve  significant 
degrees  of  human  intervention  and  judgement  are  unfeasible  in  such  simulation 
environments. 

The  SIAP  SE  TF  plans  to  pool  data  from  a  wide  range  of  test  exercises  and 
simulations  for  use  in  SIAP  assessment,  and  a  consistent  approach  to  measurement  is 
desirable.  The  automated  algorithms  preferred  in  the  simulation  community  can  be 
adapted  to  both  simulation  and  live  testing  environments,  and  represent  the  only  viable 
option  for  assignment  if  simulations  and  live  tests  are  to  be  scored  on  an  equal  footing. 
Hence,  the  use  of  an  automated  algorithm  to  compute  SIAP  attributes  will  be  the 
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standard  for  current  SIAP  assessments.  This  by  no  means  implies  that  the  SIAP  SE  TF 
discourages  other  assignment  approaches,  or  will  disregard  data  analyzed  through 
other  means.  In  fact,  comparisons  of  different  assignment  procedures  applied  to  the 
same  data  set  will  undoubtedly  prove  useful  as  the  approaches  described  in  this  report 
are  adapted  to  meet  future  needs.  For  this  reason,  multiple  analyses  using  different 
assignment  procedures  are  encouraged  whenever  this  is  a  feasible  option.  Flowever, 
until  formally  changed  or  modified,  characterization  of  performance  using  “SIAP  Metrics” 
must  be  reported  using  the  procedures  identified  in  this  technical  report  as  well  as  SIAP 
Technical  Report  2001  -001  and  2001-002. 

3.2.2  Gated  Assignment 

The  computation  of  the  SIAP  attributes  requires  distinguishing  between  tracked 
and  untracked  objects,  and  between  assigned  and  unassigned  (spurious)  tracks.  This 
is  most  immediately  obvious  in  the  attribute  of  completeness  (the  ratio  of  tracked 
objects  to  all  objects),  and  in  the  spurious  track  measure  (the  ratio  of  spurious  tracks  to 
all  tracks)  of  clarity,  but  in  fact  most  of  the  other  attributes  as  well  depend  on  a  prior 
knowledge  of  which  tracks  are  assigned.  The  assignment  procedure,  then,  must  make 
these  distinctions. 

As  already  suggested,  the  simplest  approach  to  deciding  whether  a  track  is 
assigned  or  whether  an  object  has  a  track  assigned  to  it  is  to  rule  out  implausible  track- 
object  pairings  through  gating.  Current  SIAP  assessments  will  use  a  position/velocity 
gating  criterion  (details  given  in  3.3.5),  together  with  a  qualitative  criterion  prohibiting  the 
assignment  of  a  track  to  an  object  in  a  different  environmental  category  (for  example,  an 
air  track  cannot  be  assigned  to  an  exo-atmospheric  object). 

Figure  1  illustrates  the  use  of  position  gating  (a  subset  of  position/velocity  gating) 
to  identify  an  untracked  object  (object  C). 
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Figure  1 .  Use  of  gated  assignment. 


3.2.3  Unique  Optimal  Assignment 

The  need  for  unique  (and  in  particular,  unique  optimal)  assignment  is  more 
subtle.  To  illustrate  this,  Figure  2  depicts  a  typical  situation  that  may  arise  with  tracks 
from  a  single  sensor  with  a  slight  systematic  bias  (a  very  common  occurrence),  in  which 
gating  alone  does  not  resolve  the  assignment. 
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Assume  the  use  of  an  assignment  algorithm,  which  does  not  account  for  the  bias. 
A  non-unique  assignment  would  probably  assign  both  tracks  to  object  A  (it  is  closer  to 
each  track  considered  independently),  creating  a  "local  dual."  Local  duals  do  occur,  but 
under  the  hypothesis  that  they  are  far  less  likely  on  a  single  sensor  than  is  a  systematic 
offset,  this  assignment  is  less  plausible  than  one  in  which  the  two  tracks  were  assigned 
to  distinct  objects  (this  hypothesis,  although  apparently  widely  accepted,  does  need  to 
be  tested  -  hence  the  recommendation  in  Section  5.2  of  this  report  for  experimentation 
with  alternative  algorithms).  The  non-unique  assignment  algorithm  would  thus  tend  to 
perpetuate  sensor  biases  through  the  assignment  process,  skewing  attributes  such  as 
completeness  and  clarity  in  ways  that  might  not  be  easily  traceable. 

A  unique  but  non-optimal  assignment,  such  as  for  example  a  unique  nearest- 
neighbor  assignment  procedure,  would  avoid  the  problem  just  described,  but  has  a 
different  drawback.  Unique  nearest-neighbor  assignment,  if  not  otherwise  qualified, 
depends  on  the  order  in  which  the  assignment  is  done,  which  is  an  undesirable 
characteristic  in  an  automated  program.  If  qualified  according  to  some  arbitrary 
ordering,  a  nearest-neighbor  assignment  would  be  just  as  likely  (in  the  example  in 
Figure  2)  to  make  the  intuitively  implausible  assignment  of  2-to-A,  1  -to-B,  as  the  more 
intuitive  (and  consistent  with  the  presence  of  sensor  bias)  1  -to-A,  2-to-B.  Furthermore, 
certain  algorithms  which  are  designed  to  process  the  closest  assignments  first  (so- 
called  "greedy"  algorithms),  would  always  make  the  less  plausible  choice  with  respect  to 
track  1 . 


Imposing  a  criterion  of  global  error  optimization  (that  is,  minimization  of 
cumulative  error  over  all  assigned  tracks)  on  a  unique  assignment  algorithm  is  the 
easiest  way  to  avoid  these  pitfalls.  Of  all  of  the  options  so  far  discussed,  only  a  unique 
optimal  assignment  would  consistently  make  the  most  plausible  choice  in  the  presence 
of  small  sensor  bias,  and  thus  tend  to  compensate  automatically  for  systematic  errors 
and  not  perpetuate  them  through  the  process.  A  UOA  algorithm  is  thus  proposed  for 
assigning  tracks  from  a  single  source  (for  example,  local  tracks  held  by  a  single 
participant,  or  remote  tracks  originating  from  the  same  source). 

The  situation  is  different,  however,  when  grouping  tracks  originating  from 
different  sources  (for  example,  when  assessing  local  and  remote  tracks  together).  In 
this  case,  duals  are  frequent  and  need  to  be  assessed  accurately.  Also,  local  duals 
which  cannot  be  ruled  out  through  a  locally-applied  unique  assignment  should  of  course 
be  counted  along  with  the  other  (local-remote)  duals.  This  is  the  rationale  for  assessing 
tracks  from  separate  source  nodes  independently,  and  for  the  "second  pass"  using  a 
non-unique  INNA  procedure  to  assign  the  remaining  tracks  after  all  tracked  objects  are 
accounted  for  through  the  initial  pass  with  UOA. 

3.2.4  Time  Aligning 

Time-dependent  SIAP  metrics  obtained  from  test  data  or  simulations  will  be 
evaluated  over  a  discrete  set  of  predetermined  "scoring  times."  Each  scoring  time  is 
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meant  to  be  representative  of  a  small  interval  of  time  within  a  longer  period  of 
evaluation.  In  addition,  there  may  be  certain  set  events  particular  to  individual 
participants  or  systems  at  which  evaluation  is  desired,  regardless  of  the  time  at  which 
these  events  occur  (these  are  sometimes  called  "test  article  requested"  scoring  times). 
With  the  exception  of  the  latter,  it  is  recommended  that  scoring  times  be  set  in  advance 
according  to  some  randomized  spread  throughout  the  evaluation  period,  to  avoid 
phasing  bias.  For  example,  a  requirement  could  be  imposed  of  one  scoring  time  in 
each  of  a  regularly  spaced  series  of  subintervals,  but  with  the  scoring  time  placed 
randomly  within  the  subinterval.  In  the  case  of  Monte  Carlo  simulations,  once  the 
schedule  of  scoring  times  is  determined,  the  same  schedule  (that  is,  the  same  times, 
without  additional  randomization)  should  be  used  for  all  runs.  This  consistency  between 
runs  is  important  for  statistical  analysis. 

The  scoring  times  will  not,  of  course,  generally  coincide  with  track  reporting 
times.  A  convention  is  needed  to  match  track  data  with  truth  data  at  a  particular  scoring 
time  for  purposes  of  assignment.  The  following  procedure  based  on  the  interpolation  of 
truth  data  on/y  will  be  followed.  For  assignment  of  a  local  track  at  scoring  time  tk, 
interpolate  all  truth  object  states  to  the  last  track  update  time  before  (or  equal  to)  tk,  and 
assess  the  assignment  costs  for  the  track  using  track  data  at  the  update  time.  For  the 
assignment  of  a  remote  track,  follow  an  analogous  procedure  using  instead  the  time 
stamp  in  the  track  (if  available,  as  in  the  Link  1 6  J3.6  Space  Track)  or  the  report  receipt 
time  if  there  is  no  time  stamp  (as  in  the  Link  1 6  J3.2  Air  Track).  If  it  is  independently 
known  that  a  particular  object  present  at  tk  was  not  present  at  the  update  (or  time  stamp 
or  report  receipt)  time,  then  disregard  that  track-object  pair  for  assignment  purposes 
(i.e.,  gate  out  that  track-object  pairing).  If  an  object  existed  at  track  update  time  (or  time 
stamp  or  report  receipt  time)  but  does  not  exist  at  time  tk,  then  also  disregard  that  track- 
object  pair  for  assignment  purposes. 

If  no  information  is  available  on  track  update  or  report  receipt  times,  then  there  is 
no  option  but  to  assume  that  the  track  data  is  valid  at  the  scoring  time.  This  assumption 
is  obviously  less  satisfactory  than  the  approach  just  outlined,  and  should  be  invoked 
only  as  an  alternative  to  discarding  the  track  data  entirely. 

3.3  Proposed  Assignment  Algorithm 

A  complete  description  of  the  proposed  tracks-to -truth  assignment  algorithm  is 
now  given.  The  steps  of  the  process  are  outlined  in  Section  3.3.1 .  Section  3.3.2 
addresses  the  assignment  cost  function,  and  Section  3.3.3  the  gating  criteria  to  be 
applied. 

3.3.1  Detailed  Outline  of  the  Assignment  Algorithm 

This  section  will  provide,  in  outline  form,  a  detailed  description  of  the  proposed 
track  assignment  algorithm.  It  discusses  each  track  type  (local,  remote,  etc...)  and  how 
each  track  type  is  to  be  assigned. 
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Separately  for  node  m  at  each  scheduled  scoring  time,  k, 


Step  1 :  Down  select  to  the  set  of  current  local  tracks  held  by  node  m  that  have 
precedence  for  scoring. 

Step  2:  Discard  from  consideration  at  this  scoring  time  all  remote  tracks  held  by  node  m 
which  that  node's  track  correlation  function  currently  assesses  to  be  mutual  with  node  m 
local  tracks. 

Step  3:  Down  select  to  the  set  of  current  non-mutual  remote  tracks  held  by  node  m  that 
have  precedence  for  scoring. 

Step  4;  For  J2.2  Air  PPLI  self  tracks  that  are  not  mutual  with  local  tracks, 
administratively  assign  the  PPLIs  to  the  corresponding  truth  file  based  on  unit 
identification. 

Step  5:  Separately  assign  all  retained  local  tracks  to  truth  objects,  using  a  first  pass 
through  a  GUOA  algorithm,  followed  by  a  second  pass  on  local  tracks  not  yet  assigned 
through  a  gated  INNA  (tracks  prohibited  from  all  assignments  by  gating  are  declared  to 
be  spurious). 

Step  6a:  Separately  assign  retained  non-mutual  remote  tracks  from  remote  source  node 
n  using  the  same  procedure  and  formulas  as  for  retained  local  tracks. 

Step  6b;  If  there  are  more  sources  of  retained  non-mutual  remote  tracks,  increment  n 
and  return  to  Step  6a. 

Step  7:  Merge  the  separate  assignments  into  a  complete  list  of  track-to -truth 
assignments  for  track  stores  at  node  m. 

Repeat  the  above  steps  separately  for  each  node  being  evaluated. 

The  assignment  steps  outlined  are  displayed  pictorially  in  Figure  3. 
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This  section  describes  the  formulation  of  the  assignment  cost  function  to  be  used 
in  the  first  pass  GUOA  algorithm  of  Step  5  of  Section  3.3.1 .  Recall  that  this  formulation 
applies  separately  to  each  participant  (node)  being  evaluated,  and  in  the  case  of  remote 
tracks,  to  each  source  of  tracks  held  at  each  node.  However,  so  as  not  to  encumber  the 
notation  with  additional  indices  for  the  node  and  source,  the  formulae  in  this  section  will 
be  understood  to  apply  to  tracks  held  by  a  single  participant,  originating  from  a  single 
source. 

The  proposed  UOA  algorithm  is  based  on  a  pairwise  cost  function  that  is 
effectively  a  weighted  sum  of  squared  components  of  two-dimensional  track  errors.  It  is 
the  chi-squared  (or  Mahalanobis)  distance  for  a  two-dimensional  track  with  a 
position/velocity  error  covariance  corresponding  to  a  case  in  which  errors  are 
independent  but  of  equal  magnitude  in  the  two  orthogonal  directions.  Consider  the  set 
of  tracks  held  by  the  participant  at  the  k*'^  scoring  time  (tk)  to  be  enumerated  by  an 
integral  index  i  ranging  from  1  to  the  number  of  tracks  held  (Nk).  Consider  the  set  of 
objects  present  at  the  k'^  scoring  time  to  be  similarly  enumerated  by  an  integral  index  j 
ranging  from  1  to  the  number  of  objects  (Jk).  The  unadjusted  (for  hysteresis)  cost 
function  Ck(i,,j)  for  assigning  track  i  to  truth  object  j  at  the  kth  scoring  time  is 
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(0-P)^ sinc^ -  V)^J+(0-PyJ(^  cosc^  -Vy  jJ } 


where 

Pxij,  Pyij,  Vxij,  and  Vyj  are  the  truth  object  horizontal  position/velocity 
states  in  the  track  i-centered  east-north-up  (ENU)  frame,  interpolated  to 
the  last  track  update  or  report  receipt  time,  as  described  in  Section  3.2.4, 

Si  is  the  track  horizontal  speed, 

(|)i  is  the  track  heading  measured  clockwise  from  north. 

Op  is  the  position  error  standard  deviation  in  an  orthogonal  direction, 

Y  is  the  correlation  coefficient  between  the  position  and  velocity  errors  in 
an  orthogonal  direction,  and 

(op/ov)  is  the  ratio  of  the  position  error  standard  deviation  to  the  velocity 
error  standard  deviation  in  an  orthogonal  direction. 

It  should  be  noted  that  the  truth  object  position/velocity  coordinates  Pxij,  Pyj,  Vxij,  and 
Vyj  are  not  identical  with  the  truth  object  data  used  in  the  definition  of  the  SIAP  attribute 
of  kinematic  accuracy,  as  the  latter  data  are  registered  at  the  scoring  time,  not 
interpolated  back  to  the  update  or  receipt  time. 

The  parameters  y  and  (op/ov)  weight  the  position  and  velocity  errors.  For  steady 
state  tracks  these  parameters  are  constant  with  typical  values  of  y  =  0.65  and  (op/ov)  = 

4.3  to  9.0  seconds.  The  actual  values  to  be  used  may  be  modified  after  some  initial 
experimentation.  The  quantity  op  itself  is  an  empirical  parameter  related  to  the  sensor 
characteristics  of  the  track  source  being  assessed.  It  will  generally  be  different  for  each 
sensor,  though  for  initial  assessment  purposes  the  simplifying  assumption  may  be  made 
that  Op  is  a  constant  for  all  sensors  of  the  same  type.  Values  to  use  for  op  should  be 
assessed  from  empirical  track  data. 

Appendix  B  provides  details  as  to  how  this  unadjusted  cost  function  was 
obtained,  including  a  statistical  interpretation  of  the  parameters  described  above,  and  a 
recommendation  for  the  estimation  of  opfrom  empirical  data. 

The  cost  function  given  in  equation  (1)  is  modified  to  include  a  single  time  step 
hysteresis  effect,  the  purpose  of  which  is  to  discourage  rapid  switches  in  assignment 
(though  without  eliminating  them  altogether).  The  adjusted  cost  function  for  assigning 
track  i  to  truth  object  j  at  the  kth  scoring  time  is  given  by: 

Cak(iJ)  =  Ck(i,j)  Ak(i,j)  (2) 

where  Ck(i,j)  is  the  unadjusted  cost  of  assigning  track  i  to  truth  object  j  at  scoring  time  k, 
and 
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(3) 


Jd  (0.0  <d  <  1. 0)  if  i  was  assigned  to  j  at  scoring  timek-1 
[l  .0  otherwise. 


A  default  value  of  d  =  0.15  has  been  shown  through  preliminary  experimentation  to 
produce  the  desired  hysteresis  effect  without  resulting  in  distorted  assignments.  Cost 
adjustment  with  this  default  value  should  be  regarded  only  as  a  heuristic  for  use  in  the 
absence  of  further  experimental  data.  Intuitively,  the  cost  adjustment  for  hysteresis 
should  depend  on  the  scoring  frequency.  Further  experimentation  may  show  that  it  is 
appropriate  to  base  the  dependence  of  d  upon  the  time  interval  Atk  between  the  and 
k-l^'^  scoring  times  on  an  exponential  relation  or  some  similar  functional  form. 


The  total  cost  for  an  assignment  at  time  step  k  is  then  obtained  as  follows. 
Consider  first  the  case  where  there  are  at  least  as  many  truth  objects  as  tracks.  The 
augmented  cost  matrix  is  the  NkX(Jk+1 )  matrix  of  extended  real  numbers,  rows 
corresponding  to  tracks  and  columns  to  truth  objects,  whose  (i,j)  component  is  given  by 


Cak(U) 


Cg,  if j=Jk  +  1 

<  Cak(i,j) ,  if  j  <  Jk  +1  and  gating  criteria  are  satisfied  for  the  pairing  (i,j) 
oo  otherwise. 


(4) 


where  Cg  is  a  constant  "guard  cost"  value  representing  the  cost  of  not  assigning  a  track 
to  a  truth  object.  Gating  criteria  are  discussed  in  detail  in  Section  3.3.3.  The  function  of 
the  guard  cost  is  to  ensure  that  as  many  tracks  as  possible  are  assigned  within  the 
imposed  gating  criteria,  without  the  need  for  an  explicit  representation  of  this  condition. 
The  guard  cost  value  Cg  is  related  to  the  gating  criteria,  as  will  be  made  explicit  in  the 
following  section.  An  assignment\s  represented  by  an  integer-valued,  Nk-component 
vector  ak  whose  i^'^  component  ak(i)  is  either  the  number  (from  1  to  Jk)  of  the  truth  object 
to  which  track  i  is  assigned,  or  the  number  Jk-i-1  if  the  track  is  unassigned,  with  the 
constraint  that  each  number  from  1  to  Jk  may  appear  at  most  once  as  a  component  of 
ak  (unique  assignment  constraint).  The  total  cost  CTk(ak)  for  the  assignment  ak  is  then 
given  by 


Nk - 

^1'k(ak)=  l^Cgk  (1 3k(0)  ■  (3) 

i=i 

In  the  case  where  there  are  more  tracks  than  truth  objects,  the  roles  of  tracks 
and  truth  objects  are  reversed  in  this  formulation.  That  is,  truth  objects  are  assigned  to 

tracks  (rather  than  tracks  to  objects),  and  the  augmented  cost  matrix  is  a  JkX(Nk-i-1 ) 

matrix  with  rows  corresponding  to  truth  objects  and  columns  to  tracks,  and  with  the  last 
column  containing  the  guard  costs  for  non-assignment  of  a  truth  object.  Equation  5  is 
unaltered. 
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The  optimal  assignment  is  the  assignment  ak  which  minimizes  the  total  cost.  The 
determination  of  the  optimal  assignment  is  thus  an  integer  programming  problem  (the 
number  of  assignments  being  finite,  with  at  least  one  assignment  having  finite  cost),  and 
this  problem  must  be  solved  at  each  scoring  time.  Since  there  are  no  complicated 
constraints  on  the  integral  variables  (the  components  of  ak),  the  integer  programming 
problem  is  algorithmically  straightforward,  and  a  number  of  standard  routines  are 
available  for  computing  the  optimal  solution.  The  SIAP  SE  TF  will  recommend  a 
specific  solution  program  on  the  basis  of  some  initial  estimates  of  time  and  storage 
requirements,  once  the  Common  Reference  Scenario  (CRS)  for  SIAP  evaluations  has 
been  established. 


3.3.3  Gating  Criteria 

The  exact  gating  criteria  used  may  need  to  be  modified  after  some  initial 
experimentation.  For  initial  assessments,  the  following  criteria  are  recommended. 

(1 )  A  track  must  be  for  the  same  environmental  category  as  the  truth  object: 

An  air  track  (J3.2  report)  is  only  assignable  to  a  truth  object  that  stays  aloft  by 
aerodynamic  lift  and/or  by  virtue  of  having  density  lower  than  mean  sea  level  air 
density,  or  to  an  air-to-air  missile  (AAM),  an  air-to -surface  missile  (ASM),  or  a 
surface-to-air  missile  (SAM). 

A  space  or  ballistic  missile  track  (J3.6  report)  is  only  assignable  to  a  truth  object  that 
is  in  exo-atmospheric  orbital,  sub-orbital,  or  super-orbital  trajectory,  or  to  a  ballistic 
missile  or  BMD  interceptor  object  in  any  phase  of  powered,  ballistic,  or  maneuvering 
flight,  including  a  SAM  that  is  being  operated  as  a  BMD  interceptor. 

(2)  An  air  track  i  for  which  an  altitude  state  is  provided  (and  not  declared  to  be 
unreliable)  is  only  assignable  b  a  truth  object  j  for  which  (Pzi  -  Pzj)%Rz^  <  19.5, 
(i.e.,  a  0.99999  probability  gate  in  altitude),  where  orz^  is  the  typical  variance  of  a 
track  altitude  state  for  a  sensor  of  the  type  that  generated  track  i.  Quantify  orz^ 
empirically  in  the  same  step  used  to  quantify  op/  =  opy^  =  for  the  assignment 
cost  function. 

(3)  An  air  track  i  is  only  assignable  to  truth  object  j  for  which  the  recommended  cost 
function  C(i,,j)  <  28.5  (i.e.,  a  0.99999  probability  gate  in  4  horizontal  states). 

(4)  The  same  cost  value  of  28.5  used  for  the  horizontal  gate  should  also  be  used  as  the 
guard  cost  Cg  (see  Section  3.3.2)  for  each  unassigned  track  or  untracked  object. 

3.4  Advantages  and  Disadvantages  of  the  Proposed  Procedure 

This  section  will  summarize  the  strengths  and  weaknesses  of  the  proposed 
assignment  algorithm.  A  primary  advantage  of  giving  local  mutuals  scoring  precedence 
over  remote  mutuals  is  that  local  track  data  is  always  available  in  all  legacy  air  defense 
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systems,  whereas  remote  mutual  data  is  not  always  retained  after  correlation.  Down 
selection  of  multiple  local  and  remote  tracks  facilitates  the  scoring  of  tracks  that 
operators  are  most  likely  to  use.  Administrative  assignment  of  J2.2  Air  PPLI  tracks  is 
based  on  accurate  self-identification  information.  Another  strength  of  the  proposed 
algorithm  is  separate  gated  unique  optimal  assignments  of  tracks  from  the  same  source 
tends  to  remove  source  platform  position  bias  and,  to  some  extent,  sensor  bias  from  the 
tracks-to -truth  reconstruction.  Desirable  assignment  hysteresis  across  scoring  times 
can  be  induced  through  appropriate  design  of  assignment  cost  functions.  Finally,  gating 
criteria  are  easily  modified  so  as  to  consider  factors  other  than  real-valued  distance 
measures,  offering  greater  flexibility  to  the  assignment  procedure. 

There  are  two  disadvantages  of  the  proposed  assignment  algorithm.  The  first  is, 
with  separate  assignment  of  tracks  to  truth  at  each  node,  the  same  network  track 
number  can  end  up  being  assigned  to  different  truth  objects  at  different  nodes. 

Second,  down  selection  of  local  and  remote  tracks  to  score  in  constructive  simulations 
requires  modeling  of  local-local  track  and  remote-remote  track  precedence  logic,  which 
could  be  different  at  each  type  of  node. 


4.  OTHER  RECONSTRUCTION  ISSUES 

Reconstruction  is  a  term  used  in  the  testing  community  that  refers  to  all  aspects 
of  the  process  of  constructing  track-to-truth  associations  from  data.  The  assignment 
procedure  discussed  in  Section  3  is  a  major  aspect  of  this  process,  but,  as  already 
discussed,  involves  something  of  an  idealization.  Many  other  problems  arise  in  practice 
that  may  prevent  a  smooth  application  of  this  idealized  assignment  process.  For  the 
most  part,  these  issues  have  not  as  of  yet  been  specifically  addressed  by  the  SIAP  SE 
TF.  The  following  sub-sections  provide  a  brief  discussion  of  some  of  the  reconstruction 
issues  that  may  arise.  One  recommendation,  based  on  input  from  representatives  of 
the  joint  testing  community,  is  described  in  Section  4.1 .  It  may  be  necessary  for  the 
SIAP  SE  TF  to  provide  additional  specific  recommendations  on  other  points,  if  evidence 
of  serious  difficulties  with  test  data  should  emerge  as  the  assessment  process  matures. 

4.1  Absence  of  Data 

For  whatever  reason,  the  data  for  a  particular  participant  in  a  test  event  or 
exercise  may  not  be  recorded  or  it  may  be  lost.  Because  the  calculations  for  the  SIAP 
attributes  are  based  on  recorded  track  data  from  all  participants,  the  absence  of  data  is 
an  issue,  and  how  these  cases  are  handled  needs  to  be  addressed.  One  does  not 
simply  want  to  discard  the  data  from  all  of  the  units  just  because  one  participant’s  data 
cannot  be  included  in  the  attribute  calculations.  The  following  presents  a  recommended 
procedure  for  computing  averages  of  SIAP  attributes  without  penalizing  a  participant  for 
the  lack  of  data.  The  variable  defined  as  the  number  of  objects  at  time  t  is  allowed  to 
depend  on  the  participant  m,  and  is  now  denoted  Jm(t).  As  long  as  data  reported  by 
participant  m  is  recorded,  Jm(t)  is  the  same  number  for  all  m,  J(t).  Flowever,  if  the 
reporting  of  data  from  participant  m  is  interrupted,  Jm(t)  should  be  set  to  zero,  producing 
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an  analytical  situation  in  which  it  is  as  if  participant  m  were  not  involved  in  the  test  at 
that  time. 

This  convention  attempts  to  offset  data  dropouts  that  may  be  encountered  due  to 
problems  with  the  testing  architecture  or  process,  as  opposed  to  problems  with  the 
network(s)  being  tested.  Equations  for  the  IADS  roll-ups  of  the  attribute  measures  can 
be  adjusted  accordingly. 

As  an  example,  consider  the  attribute  of  completeness,  and  assume  that  there 
are  three  participants  in  the  current  scenario  with  four  objects  in  the  AOI  from  time  zero 
to  time  three.  Participant  1  tracks  and  reports  all  four  objects  at  each  time  period. 
Participant  2  tracks  and  reports  three  objects  at  each  time  period,  and  a  fourth  at  times 
zero,  one,  and  three.  Participant  3  tracks  and  reports  exactly  three  objects  at  each  time 
period;  however,  no  data  is  recorded  at  times  two  and  three.  Recall  the  completeness 
formula  (SIAP  SE  TF  Technical  Report  2001  -001 , 

c„(t)=^t!^ioo%.  (6) 

J^tj 

The  IADS  roll-up  is  adjusted  from  the  usual  average  over  time  steps  and  participants  to 
the  modified  average 


C  = 


M  tend 

I  I  jT„(t) 

m  =1  t  start 
M  tend 

I  £  j-(t) 

m  =  1  tstart 


1 00  % 


(7) 


where  Jm(t)  is  the  adjusted  object  count  as  defined  above  (for  definitions  of  other 
variables  see  Appendix  A).  The  roll-up  completeness  C  following  the  unadjusted 
averaging  formula  (SIAP  SE  TF  Technical  Report  2001-001 )  for  the  example  would  be 

^  (4+4+4+4)+(4+4  +  3+4)+(3  +  3  +  0  +  0) 

(4  +  4  +  4  +  4)  +  (4+4+4+4)+(4  +  4  +  4  +  4)  ' 


thus  reckoning  completeness  as  if  participant  3  were  tracking  no  objects  during  the 
periods  of  data  dropout.  Using  the  proposed  adjusted  formula  (5),  completeness 
becomes 

^  (4  +  4+4+4)  +  (4+4+3  +  4)  +  (3  +  3  +  0  +  0) 

'^-(4  +  4  +  4  +  4)  +  (4+4  +  4  +  4)  +  (4  +  4+0  +  0)“  '  ' 

A  similar  adjustment  can  be  made  with  any  lADS-averaged  metric.  It  is  important 
not  to  overuse  this  approach,  as  there  is  a  danger  of  skewing  the  averaging  results  to 
an  extent  that  they  can  no  longer  be  understood  intuitively.  If  the  absence  of  data  is  a 
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persistent  problem,  then  a  safer  approach  would  be  to  present  a  sample  instantaneous 
data  set  in  timeline  form  (such  as  depicted  in  Figure  4)  and  show  where  the  problem 
lies. 


A  problem  more  difficult  to  remedy  is  the  dropout  of  truth  data  for  particular 
periods  of  time.  SIAP  Metrics  cannot  be  calculated  without  accounting  for  truth  objects. 
Unless  an  appropriate  substitute  can  be  found  for  the  data  intended  to  be  collected  (for 
example,  perhaps  a  reliable  sensor  held  good  track  data  on  all  objects  for  the  critical 
period),  the  only  safe  remedy  is  to  restrict  the  period  of  evaluation  to  times  for  which  the 
truth  data  is  available.  Fortunately,  it  appears  that  the  use  of  PPLI  data  for  truth  in 
many  recent  joint  exercises  has  reduced  the  severity  of  this  problem  (though  without 
completely  eliminating  it),  at  least  for  friendly  truth  objects. 

4.2  Time  Aligning 

Time  aligning  has  already  been  discussed  in  the  context  of  the  tracks-to -truth 
assignment  algorithm.  However  there  may  be  other  issues  involving  time  that  arise 
from  the  specific  details  of  various  live  tests.  If  time  offset  and/or  scale  factor  bias  is 
evident  in  any  metric  data  that  are  to  be  treated  as  truth  data  in  the  tracks-to -truth 
assignment,  either  those  data  can  be  discarded  (and  the  consequences  of  their  loss 
accepted)  or  attempts  can  be  made  to  remove  the  time  bias  so  that  the  data  can  still  be 
used.  For  example,  under  certain  circumstances  this  could  be  done  by  first  estimating 
and  compensating  the  time  and  measurement  biases  of  one  of  the  participating  sensors 
using  other  truth  data  that  can  be  trusted,  and  then  solving  for  the  time  offset  and  scale 
factor  bias  in  the  corrupted  truth  data  that  minimizes  the  sum  of  distances  between  that 
data  and  the  measurements  or  track  state  estimates  of  the  registered  sensor.  Except 
as  an  indirect  means  to  remove  time  bias  in  corrupted  truth  data  (or  as  part  of  a 
separate  assessment  to  characterize  as-is  sensor  biases  and  the  benefit  of  correcting 
them),  usually  no  attempt  should  be  made  to  remove  time  or  other  biases  in  track  data 
that  are  to  be  scored. 

4.3  Formation  Tracking  Considerations 

The  use  of  formation  tracking  introduces  many  complications,  and  an  appropriate 
assessment  procedure  has  not  yet  been  worked  out.  Formation  tracking  and 
assessment  issues  are  currently  under  study  by  a  SIAP  SE  TF  working  group.  The 
recommendations  of  this  working  group  will  be  addressed  in  another  technical  report, 
and  to  the  extent  that  they  influence  the  near-term  SIAP  assessment  approach,  will  be 
incorporated  into  future  versions  of  this  report. 

There  is,  however,  an  immediate  need  to  set  down  a  convention  for  scoring  SIAP 
attributes  and  track-based  MOPs  in  tests  in  which  formation  tracking  is  used.  U.S.  Joint 
Forces  Command  (USJFCOM)  has  recently  provided  a  clarification  of  TAMD  CRD 
requirements  for  formation  tracking,  which  sets  the  long-term  objective  that  "each  object 
should  have  a  unique  track  identifier  and  associated  characteristics"  (CINCUSJFCOM 
J8  Memorandum  of  29  June,  2001).  In  keeping  with  this  long-term  objective,  the  SIAP 
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SE  TF  will  base  SIAP  attribute  measures  and  appropriate  MOPs  on  a  one-object  per 
track  basis,  with  no  special  allowances  made  for  formation  tracks.  Formation  tracks  will 
be  treated  as  any  other  tracks  in  the  assignment  procedure  of  Section  3,  and  scored 
accordingly.  Special  MOPs  have  been  introduced  (cf.  SIAP  SE  TF  Technical  Report 
2001  -002)  to  measure  the  extent  of  formation  track  use  in  a  testing  situation,  and  these 
MOPs  provide  a  gauge  of  the  effect  of  this  one  object  per  track  scoring  basis  on 
assessment  results  for  formation  trackers. 

Exceptions  to  this  scoring  policy  may  be  in  order  for  ballistic  missile  tracks  in 
certain  situations.  USJFCOM  explicitly  recognizes  this  in  the  statement,  "the  unique 
issues  of  dealing  with  ballistic  missile  tracks  will  differ  in  this  regard  and  may  require 
further  study"  (CINCUSJFCOM  J8  Memorandum  of  29  June,  2001).  The  SIAP  SE  TF 
has  elsewhere  acknowledged  (cf.  SIAP  SE  TF  Technical  Report  2001  -001 )  that  ballistic 
missile  defense  (BMD)  issues  may  require  reconsideration  of  some  elements  of  the 
proposed  approach  to  SIAP  metrics,  which  may  include  the  implementation  proposals 
made  in  this  report.  Recommendations  pertaining  to  BMD  have  been  deferred  pending 
further  study  by  the  SIAP  SE  TF  and  other  organizations  involved. 

4.4  Dimensionality  in  Accuracy  Calculations 

Not  all  tracking  systems  work  in  three  dimensions.  Even  for  systems  capable  of 
receiving  three-dimensional  information,  it  is  still  possible  that  only  two-dimensional  data 
is  being  recorded  in  a  test  situation  (for  example,  some  data  collection  formats  do  not 
use  a  height  rate  field).  This  presents  some  ambiguity  in  the  definitions  of  kinematic 
accuracy  measures,  as  already  noted  in  SIAP  SE  TF  Technical  Report  2001-001 . 

The  current  definitions  allow  the  test  designer  to  weight  the  vertical  position  and 
velocity  accuracy  measures  by  some  factor  w  between  zero  and  one,  to  scale  out 
vertical  errors  which  a  2D  system  is  intrinsically  incapable  of  improving,  and  which 
perhaps  cannot  be  reliably  estimated  anyway.  In  most  cases,  w  will  be  set  to  exactly 
zero  or  exactly  one,  depending  on  the  dimensionality  of  the  current  data.  This  allows  a 
fair  comparison  among  sensors  of  the  same  dimensionality,  but  it  is  not  clear  whether 
either  choice  will  give  useful  accuracy  measures  when  the  sensors  are  mixed  (which  is 
in  fact  quite  typically  the  case  for  current  sensor  networks).  There  is  the  possibility  of 
using  intermediate  values  of  w  to  decrease  the  significance  of  the  third  dimension,  while 
maintaining  the  third  measure,  but  the  SIAP  SE  TF  has  not  agreed  to  any  standard  for 
setting  the  value. 

There  needs  to  be  further  exploration  of  the  possible  uses  of  the  kinematic 
accuracy  measure  among  joint  systems  subject  area  experts  before  this  issue  can  be 
resolved  entirely.  For  present  assessments  concerning  only  air  vehicle  tracks,  the  SIAP 
SE  TF  will  compute  the  kinematic  accuracy  measure  on  a  2D  basis  (w=0  for  all  tracks 
and  all  participants),  but  recommends  the  use  of  3D  information  where  available  and 
appropriate  for  the  computation  of  MOPs  in  root  cause  analysis.  Again,  this  approach  is 
not  expected  to  be  adequate  for  the  BMD  component  of  the  SIAP,  for  which  full 
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dimensionality  (3D  or  6D)  in  accuracy  is  essential.  The  approach  will  therefore  be 
updated  with  future  versions  of  this  report  as  BMD  recommendations  are  included. 


The  proposed  assignment  algorithm  of  Section  3  is  basically  two-dimensional, 
but  allows  for  altitude  gating  on  appropriate  tracks.  The  dimensionality  issue  is  raised 
here  because  the  reconsideration  of  the  kinematic  accuracy  definition  may  lead  to  a 
reconsideration  of  the  assignment  procedure  in  this  regard. 

4.5  Test  Design  Issues 

The  SIAP  SE  TF  may  need  to  look  at  specific  problems  associated  with  testing 
design  and  make  recommendations  to  help  bring  test  designs  closer  into  line  with  SIAP 
assessment  objectives. 

5.  EXCURSIONS  AND  OTHER  APPROACHES 

The  proposals  for  current  assessments  do  not  represent  all  of  the  data  that  will 
have  to  considered  throughout  the  SIAP  system  engineering  process.  This  section 
provides  other  possible  methods  for  treating  SIAP  data,  including  a  list  of  alternative 
track  bases  from  which  to  select  candidate  tracks  as  well  as  corresponding  alternative 
scoring  approaches.  Experimentation  with  these  approaches  and  excursions  is 
encouraged.  However,  until  formally  changed  or  modified,  characterization  of 
performance  using  “SIAP  Metrics”  must  be  reported  using  the  procedures  identified  in 
this  technical  report  as  well  as  SIAP  Technical  Report  2001  -001  and  2001  -002. 

5.1  Alternative  Track  Bases 

Many  aspects  of  the  proposed  metrics  implementation  approach  can  be  easily 
adapted  to  any  set  of  tracks.  As  mentioned  in  Section  2,  the  approach  to  measurability 
for  current  assessments  is  not  the  only  one  that  should  be  examined.  Any  data 
collected  from  other  track  bases  may  be  useful  for  SIAP  system  engineering  in  various 
ways,  and  in  particular  may  reveal  difficulties  with  or  suggest  improvements  to  the 
proposed  approach.  Furthermore,  the  relative  importance  of  different  track  bases  for 
warfighting  capability  assessments  can  be  expected  to  change  as  SIAP  system 
engineering  evolves. 

This  following  is  a  (nonexhaustive)  list  of  alternative  track  data  sets  which  are  of 
interest  in  this  regard. 

•  Platform  track  stores  (current  approach) 

•  Link  1 6  tracks  only 

•  Link  1 1  tracks  only 

•  Link  22  tracks  only 

•  CEC/JCTN  composite  tracks  only 

•  PPLI  tracks  only  (subset  of  link  tracks) 

•  Local  sensor  tracks 
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Since  a  number  of  excursions  may  be  pursued  simultaneously  in  different  testing 
environments,  it  is  important  to  record  the  track  data  sets  actually  used  in  each 
excursion.  Therefore,  the  SIAP  SE  TF  will  require  as  a  general  policy  that  any 
assessments  done  under  its  sponsorship  be  submitted  with  a  record  defining  the  track 
data  used  at  the  participant  level.  Short  descriptive  statements  (such  as  appear  in  the 
list  just  noted)  are  adequate,  provided  differences  in  scoring  policy  among  different 
participants  are  noted  as  well. 

Table  2  summarizes  possible  scoring  excursions  for  each  of  these  possible  track 
bases,  and  the  corresponding  adjustments  to  the  assignment  procedure  required. 


Track  Base 

Scoring  excursion 

Platform  track  stores 
(current  approach) 

Local  mutuals  have  precedence  for 
scoring 

Remote  mutuals  have  precedence  for 
scoring 

A  table  defines  local  versus  remote 
precedence  for  scoring. 

Link  16  tracks 

The  set  of  local  R2  tracks,  remote  J3.2  Air 
Tracks,  and  J2.2  Air  PPLIs 

The  set  of  local  R2  tracks  and  remote  J3.2 

Air  T  racks 

Exclusively  J2.2  Air  PPLIs 

Link  1 1  tracks 

Can  include  variants  for  separate 
assignment  at  each  node  and  for  global 
assignment  of  network  tracks 

Link  22  tracks 

Can  include  variants  for  separate 
assignment  at  each  node  and  for  global 
assignment  of  network  tracks 

CEC/JCTN  composite  tracks 

Can  include  variants  for  separate 
assignment  at  each  node  and  for  global 
assignment  of  network  tracks 

Local  sensor  tracks 

Local  tracks 

Table  2.  Alternative  track  bases  and  scoring  excursions. 
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5.2  Alternative  Approaches  to  Assignment 


Recall  in  Section  3  that  the  SIAP  SE  TF  encourages  alternative  approaches, 
partly  as  a  way  of  gauging  the  effectiveness  of  the  proposed  procedure. 

The  procedure  as  outlined  in  Section  3  can  easily  be  modified  to  accommodate 
other  scoring  preferences.  For  example,  reasons  have  been  given  why  local/mutuals 
have  been  given  scoring  precedence;  however,  it  may  be  worthwhile  to  explore  the 
impact  of  scoring  remote/mutuals  over  local/mutuals.  Remote  track  data  will  almost 
certainly  increase  in  both  accessibility  and  importance  with  future  SIAP  improvements. 

To  accommodate  this  variant,  it  is  only  necessary  to  change  the  roles  of  local/mutual 
and  remote/mutual  tracks  in  the  assignment  procedure  outlined  in  Section  3.3  (i.e., 
local/mutuals  are  omitted  in  the  pass  through  local  tracks,  while  remote/mutuals  are 
assigned  just  as  any  other  remote  tracks). 

It  is  also  advisable  to  experiment  with  variants  of  the  time  aligning  proposal  of 
Section  3.2.4.  The  proposed  procedure  bases  assignment  on  track  and  truth  data  at 
update  (or  report  receipt)  times.  This  approach  obviates  the  need  to  extrapolate  track 
data,  and  is  thus  expected  to  avoid  the  introduction  of  additional  extrapolation  errors 
into  the  assignment  procedure.  Furthermore,  the  use  of  the  proposed  time  aligning  has 
been  implicitly  assumed  in  the  derivation  of  some  details  of  the  cost  function  (see 
Appendix  B),  and  is  convenient  in  that  respect.  However,  there  are  some  concerns  over 
the  algorithmic  complexity  of  this  approach,  which  may  in  principle  require  a  truth  data 
interpolation  for  every  possible  truth-track  pairing  at  every  time  step.  Also,  there  are 
concerns  that  anomalies  in  some  of  the  attribute  measures  may  result  if  there  is 
significant  "coasting"  of  tracks  (retention  of  tracks  in  the  absence  of  data  updates),  as 
coasted  tracks  will  continue  to  be  assigned  as  they  would  have  been  on  the  basis  of 
their  last  update  times.  To  trace  the  possible  effects  of  track  coasting,  an  alternative 
time  aligning  scheme  based  on  extrapolation  of  track  data  to  the  current  scoring  time 
(rather  than  interpolation  of  truth  data  to  update  or  receipt  times)  should  be  explored. 
Differences  will  very  likely  be  observed  in  the  measures  of  completeness  and  spurious 
tracks,  since  the  alternative  scheme  will  have  a  greater  tendency  to  declare  coasted 
tracks  to  be  spurious.  However,  it  is  difficult  to  foresee,  without  experimentation,  which 
time  aligning  scheme  will  yield  more  intuitively  meaningful  or  analytically  useful  values 
for  these  metrics,  or  how  great  the  differences  are  in  practice.  Some  initial 
experimentation  will  also  be  needed  to  determine  whether  there  are  serious  algorithmic 
complexity  or  inefficiency  issues  with  the  proposed  approach.  It  should  be  kept  in  mind 
as  well  that  the  viability  of  any  time  aligning  proposal  for  a  live  test  evaluation  is  to  some 
extent  dictated  by  the  quality  and  nature  of  the  time  data  available,  and  by  the  way  the 
data  is  formatted  and  stored.  The  method  proposed  in  Section  3.2.4  is  presently 
believed  to  be  suitable,  at  least  for  initial  experimentation,  in  the  context  of  live  tests  and 
simulations  planned  over  the  next  year  or  two,  but  future  experience  may  lead  to 
different  recommendations. 

One  possible  problem  with  the  proposed  assignment  algorithm,  noted  in  Section 
3.4,  is  the  possibility  of  a  counter-intuitive  assignment  of  the  same  track  to  different 
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objects,  depending  upon  the  participant  being  assessed.  Although  this  violates 
common  sense,  it  is  not  clear  whether  these  multiple  assignments  will  be  numerous,  or 
whether  they  will  skew  the  calculation  of  SIAP  attribute  measures  significantly.  One 
way  of  examining  whether  this  issue  represents  a  genuine  drawback  to  the  approach  is 
to  compare  assessments  based  on  the  proposed  assignment  with  re-calculations  using 
a  different  approach  -  one  for  which  a  track  is  forced  to  be  assigned  to  the  same  object 
for  all  participants.  The  latter  condition  is  satisfied  by  another  variant  of  the  proposed 
assignment  algorithm,  in  which  remote  tracks  are  always  assigned  according  to  the 
assignment  selected  at  the  source  node.  That  is,  all  local  tracks  are  assigned  as  usual, 
and  remote  tracks  inherit  the  assignments.  This  variant  carries  its  own  disadvantages 
(possibilities  for  peculiar  scoring  in  cases  of  poor  data  registration),  but  it  would  be 
instructive  to  set  these  against  the  disadvantages  of  the  proposed  procedure. 

Finally,  as  mentioned  in  Section  3.2.1 ,  automated  assignment  algorithms  have 
not  been  used  uniformly  throughout  the  joint  testing  community,  and  it  is  reasonable  to 
question  whether  the  standardization  of  automated  assignment  proposed  in  this  report 
offers  any  improvement  over  traditional  procedures  (many  relying  partly  on  human 
intervention).  The  best  way  to  answer  this  question  would  be  to  pursue  all  approaches 
to  the  extent  possible.  Comparison  in  simple  cases  between  manual  or  semi- 
automated  procedures  with  proven  reliability  and  fully  automated  algorithms  such  as  the 
one  proposed  in  this  report  are  an  excellent  means  of  verifying  the  effectiveness  of  the 
latter  in  live  test  situations,  as  well  as  suggesting  possible  improvements.  If  frequent 
and  severe  discrepancies  in  assessments  were  obtained  through  different  assignment 
procedures,  this  would  naturally  be  grounds  for  serious  reconsideration  of  the  proposed 
approach. 


6.  CONCLUSIONS  AND  RECOMMENDATIONS 

These  proposals  for  SIAP  metrics  implementation,  together  with  the  detailed 
definitions  of  SIAP  metrics  provided  in  other  SIAP  SE  TF  publications,  should  satisfy  the 
most  important  immediate  objective  for  SIAP  assessments  -  that  of  providing  a 
consistent  approach  to  measuring  the  quality  of  the  current  SIAP.  Particular  attention 
has  been  paid  to  establishing  consistency  in  data  assessment  between  live  exercise 
and  test  evaluation  on  the  one  hand,  and  modeling  and  simulation  on  the  other. 

It  is  recommended  that  the  implementation  proposals  in  this  report  be  applied  to 
both  simulation  and  live  exercise  data.  It  is  recognized  that  the  implementation 
strategies  proposed  will  be  modified  as  the  methodology  is  used  and  improvements  are 
identified. 
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APPENDIX  A 


Critical  Variables  for  SIAP  Attributes  Impiementation 

This  Appendix  offers  a  systematic  classification  scheme  for  all  variables  that  are 
directly  involved  in  the  mafhemaf/ca/ derivation  of  the  SIAP  attribute  measures.  The 
assumption  is  made  in  this  treatment  that  the  measurability  and  reconstruction  problems 
(including  a  complete  tracks-to-truth  assignment)  have  been  resolved  for  a  hypothetical 
situation  under  assessment. 

One  point  of  this  exercise  is  to  demonstrate  exactly  how  the  assignment  function 
(the  formal  mathematical  representation  of  the  assignment,  of.  Section  3.1  of  the  main 
text)  enters  into  the  derivation  of  the  SIAP  attribute  measures.  Another  is  to  identify  a 
minimal  set  of  additional  intermediate  variables  that  must  be  extracted  from  track  data 
and  truth  to  complete  the  derivation.  Finally,  Appendix  A  concludes  with  a  table  (Table 
A-1)  which  can  serve  as  a  convenient  reference  for  all  formulae  involved  in  the 
definitions  of  the  SIAP  attributes  themselves  (adapted  from  SIAP  SE  TF  Technical 
Report  2001-001). 

The  overall  scheme  is  suggested  by  Figure  A-1 . 


SIAP  Attributes 


Truth  Data 


Tracking  and  ID  Performance 


Figure  A-1 .  SIAP  attributes  implementation. 
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All  variables  falling  into  the  three  "boxes"  leading  directly  to  the  SIAP  attributes  will  be 
explicitly  identified,  and  the  mappings  suggested  by  the  green  (filled)  arrows  will  be 
explicitly  defined. 

Critical  Variables 

The  critical  variables  consist  of  the  assignment  function  itself  (including  a 
specification  of  its  domain,  which  effectively  identifies  the  track  data  set),  some  truth 
variables  not  directly  represented  in  the  tracks-to-truth  assignment,  and  a  reduced  list  of 
MOPs  and  attributes  (called  "track-specific"  metrics  here)  which  are  directly  obtainable 
from  position,  velocity,  and  ID  data  on  each  track,  the  truth  data,  and  the  assignment 
function. 


Truth  Variables 

J(t)  -  total  number  of  objects  at  time  t 
M  -  total  number  of  tracking  participants 
[tstart,  tend]  “  interval  of  evaluation 
Tj  -  total  time  of  flight  of  object  j 


Assignment  Function 

N  -  total  number  of  track  numbers  (Central  Track  Store  Locators,  or  CTSLs)  ever 
occurring 

Dc{1...N}X{1  ...M}X[tstart,tend]- domain  of  assignment -the  set  of  triples  (n,m,t), 
where  n  is  a  track  held  by  participant  m  at  time  t  (each  CSTL  being  indexed 
by  a  positive  integer  n  from  1  to  N) 

F:D?  Z 

F(n,m,t)  -  assigns  to  each  track  at  each  time  t,  a  positive  value  representing  its 
assigned  object  j  (or  a  value  of  0  as  a  convention  for  indicating  that  the  track 
is  unassigned) 


fn,m(t)  =  F(n,m,t) 
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Track-Specific  Attributes 


Track  Position  Accuracy  [PAn,m(t)]  -  (possibly  weighted)  distance  from  position  of 
track  n  to  position  of  object  j  evaluated  at  scoring  time  t,  where  track  n  is 
assigned  to  object  j  at  time  t. 

Track  Velocity  Accuracy  [VAn,m(t)]  -  (possibly  weighted)  difference  of  velocity  of 
track  n  and  velocity  of  object)  evaluated  at  scoring  time  t,  where  track  n  is 
assigned  to  object  j  at  time  t. 

Track-Specific  MOPs 

Track  ID  Assessment  [ID  n,m(t)]  -  an  indication  of  whether  the  ID  is  incorrect  (-1 ), 
unknown  (0),  or  correct  (+1 ),  for  an  assigned  track  n  held  by  participant  m,  at 
scoring  time  t. 

Track  Commonality  [NCn(t)]  -  a  boolean  variable  (=1  if  yes,  =0  if  no),  describing 
whether  track  n  is  held  by  all  participants  at  scoring  time  t,  within  some  given 
constraints. 


Derived  Variabies 

The  derived  variables  include  all  variables  used  in  SIAP  SE  TF  Technical  Report 
2001  -001 ,  other  than  those  representing  truth  data  and  the  attribute  measures 
themselves.  They  are  explicitly  derivable  from  the  critical  variables  listed  above,  as  the 
formulae  show.  Once  these  variables  are  determined,  the  attribute  measures  can  be 
calculated  directly  from  their  definitions  (which  follow,  in  Table  A-1 ). 

Notation:  |A|  =  cardinality  of  set  A 
^(A)  =  measure  of  set  A 


Nc(t)  -  the  number  of  assigned 
tracks  held  by  all  participants  at 
time  t 

Ns(t)  -  the  number  of  assigned 
tracks  held  by  at  least  one 
participant  at  time  t 

Nm(t)  -  the  number  of  tracks  held  by 
participant  m  at  time  t 


=  lNC„(t) 

n=1 


=  |{n|3m,f,^(t)>0}| 

=  |{n|(n,m,t)eD}| 
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Dm(t)  -  set  of  assigned  tracks  held 
by  participant  m  at  time  t 

NAm(t)  -  the  number  of  assigned 
tracks  held  by  participant  m  at 
time  t 

lj,n,m  -  set  of  times  for  which  track  n 
held  by  participant  m  is  assigned 
to  object  j 

Tj,n,m  -  the  total  time  of  track  n  on 
object  j  by  participant  m 

jTm(t)  -  the  number  of  objects  being 
tracked  by  participant  m  at  time  t 

lj,m  -  set  of  times  for  which  a  track 
held  by  participant  m  is  assigned 
to  object  j 

TTj,m  -  the  total  time  object  j  is 
tracked  by  participant  m 


=  {n|fn,m(t)>0} 


=  {j|j^0A3n,f, 
=  {t|3n,f,,,(t)  = 


NUj,m-  the  minimum  number  of 
tracks  needed  to  cover  object  j 
over  TTj,m 


TLj,m- the  total  time  of  the  longest  =  max(Tjp^) 

track  on  object  j  by  participant  m 


Rj,m  -  number  of  excess  tracks 
(track  number  changes)assigned 
to  object  j  by  participant  m 


Rm  -  average  number  of  excess 
tracks  assigned  by  participant  m 


R  -  roll-up  average  of  Rm 


(NU,,„-1) 


TT 


j,m 


Dj,m(t)  -  set  of  tracks  held  by  =  {n|f^^(t)  =  j } 

participant  m  and  assigned  to 
object  j  at  time  t 


.,(t)=i}| 

j} 

,..N);lA|=K]ArYli,„„  = 
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JCm(t)  -  number  of  objects  tracked 
by  participant  m  with  correct  ID  at 
time  t 

Jlm(t)  -  number  of  objects  tracked  by 
participant  m  with  incorrect  ID  at 
time  t 

JUm(t)  -  number  of  objects  tracked 
by  participant  m  with  unknown  ID 
at  time  t 

JAm(t)  -  number  of  objects  tracked 
by  participant  m  with  ambiguous 
ID  at  time  t 


=  {j|D.Jt)#OA[vnsD,,Jt),ID„(t)  =  l]}|* 
=  {j|D,Jt)#OA[vnsD.„(t),ID„(t)=-l]}| 
=  {i|D(Jt)#0A[vneDj„,(t),ID„„(t)=0])|‘ 


*subject  to  modification  in  accordance  with  CRS 
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Xm(t) 


JTjt) 

J(t) 


NAjt) 

JTjt) 


■•ena 

£jT,(t) 

tstarl _ 

Vnri 

£j(t) 


^end 

ENA„(t) 

^start 

^end 

£jTn.(t) 


‘end 

1  M 


_y  lita 

\ 


EJ(t) 


^^end 

£NA^(t) 

■I  M  ^  ' 

J_y'  J^art _ 


Nn.(t)-NA„(t)  t  " 
N™(t)  ; 


E[N„(t)-NA„(t)]  „  EK(t)-NAjt)] 

^start _  _ \  ’  ^start _ 

iTNj,)  irN„(,) 


o 

ETL, 


u 

J_y'  _H _ 


£pA„jt)  E  EPA„A,(t)  ,  „  E  EPA„A.(t) 

PA  ne^t)  t3i,rtreD^(t) _  _}_Y  IstartreDm(t) _ 

iCNA^d)  iCNA^d) 


Range  Target 


0  =  C=100%  100% 


A  =  1 


0  =  S  =  1 00% 


LT>0 


0  =  LSt=100%  100% 


PA  =  0 


Table  A-1 .  Formulae  for  SIAP  attributes. 
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Xm(t) 

Xm 

IVA„„(t) 

neD,„(t) 

^end 

E  EVAnn,(t) 

tstartrED„,(t) 

NAjt) 

E 

< 

jTjt)-JUjt) 

£[jTjt)-Ju,,(t)] 

^start 

JTjt) 

£jT.(t) 

^start 

Jcjt) 

^end 

Ejc„(t) 

^start 

JTjt) 

^snri 

£jTjt) 

^  Start 

jAm  (t) 

E 

< 

jTm(t) 

£jT,(t) 

^start 

CM(t)*  = 

NS(t) 


£NC(t) 
QM*  =  - 

^snri 

£NS(t) 
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APPENDIX  B 


Further  Details  of  the  Assignment  Cost  Function 

This  appendix  provides  a  derivation  of  the  assignment  cost  of  Section 
3.3.2.  In  particular,  it  is  shown  that  equation  1  can  be  derived  through  the 
application  of  several  simplifying  assumptions  to  a  more  general  (more 
idealized)  formula.  The  more  general  cost  function  requires  knowledge  of 
covariance  error  data  at  each  scoring  time.  Because  such  data  may  not  always 
be  reliable  (or  available),  the  idealized  cost  function  is  not  as  easily  implemented 
as  the  practical  cost  function  prescribed  in  Section  3.3.2. 

The  general  cost  function  (and  the  approach)  described  in  this  appendix 
can  serve  as  a  starting  point  for  experimentation  with  alternative  assignment 
algorithms. 

Near-Ideal  Unadjusted  Cost  Function 

If  full,  accurate,  track  state  error  covariance  matrices  were  available  for  all 
tracks  ,  then  a  very  good  unadjusted  cost  function  would  be 

Ck(U)  =  AXi/zPij-'AXij  -G(n,a)  (B-1) 

where 

Axij  =  Xi  -  Xj  for  track  state  column  vector  Xi  and  truth  state  column  vector 
Xj,  both  consisting  of  n  comparable  states, 

ZPij  =  (Pi  +  Pj)  and  is  an  nxn  matrix  for  track  i  error  covariance  Pi  and 
truth  j  knowledge  error  covariance  Pj,  (in  simulations  where  truth  is  known 
exactly,  Pj  =  0), 

G(n,  a)  is  the  a  percentile  value  of  a  chi-squared  statistic  with  n  degrees  of 
freedom. 

The  gate  value  for  the  assignment  is  zero,  because  G(n,  a)  is  in  the  cost. 

This  formulation  would  allow  normalized  assignment  of  tracks  having 
different  dimensionality,  such  as  horizontal-only  four-state  tracks,  full 
position/velocity  six-state  tracks,  and  full  position/velocity/acceleration  nine-state 
tracks.  It  would  allow  use  of  all  the  information  in  a  local  track  file  together  with 
the  more  limited  available  information  in  a  network  track.  Unfortunately,  full  error 
covariance  information  is  not  usually  available  for  all  tracks  and,  where  it  is  be 
available  (e.g.,  some  local  tracks),  it  may  not  be  credible. 

The  adjusted  cost  Cak(iJ)  at  scoring  time  k  for  assigning  track  i  to  truth 
object  j  is 


Cak(iJ)  =  Ck(i,j)  Ak(i,j) 


(B-2) 
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where  Ck(i,j)  is  the  unadjusted  cost  of  assigning  track  i  to  truth  object  j  at  scoring 
time  k, 


fd^  (0.0  <di  <  1.0)if  CJiJ)  >  0  and  i  was  assigned  to  j  at  scoring  timek-1 


Ak(U) 


(1 .0  <  dj)  if  Ck  (i,  j)  <  0  and  i  was  assigned  to  j  at  scoring  time  k  - 1 


[1 .0  otherwise. 


(B-3) 


Practical  Unadjusted  Assignment  Cost  Function 

When  considering  aerospace  vehicle  tracks,  as  already  noted,  full  credible 
track  error  covariance  will  generally  not  be  available.  For  single-sensor  tracks  for 
which  the  source  sensor  is  collocated  with  the  participant,  the  track  error 
covariance  matrix  may  be  imputed  from  track  quality  on  the  basis  of  assumptions 
regarding  the  orientation  of  principal  axes.  For  some  remote  tracks  however, 
there  is  insufficient  information  from  which  to  determine  principal  directions  (e.g., 
the  track  is  a  composite  track  or  the  location  of  the  source  sensor  is  unknown), 
so  it  is  not  even  possible  to  impute  the  full  error  covariance.  Furthermore, 
remote  tracks  may  only  include  latitude/longitude  and  track-centered  local 
horizontal  velocity  states  (altitude  can  be  marked  as  unreliable  and  J3.2  has  no 
altitude  rate  field). 

On  the  other  hand,  because  they  may  assume  reporting  responsibility, 
tactical  command  and  control  systems  must  be  able  to  represent  air  vehicle  local 
tracks  states  in  latitude/longitude  and  track-centered  local  horizontal  velocity 
states.  Also,  truth  data  can  be  represented  in  any  convenient  coordinate  frame 

Because  for  air  vehicle  tracks  full  error  covariance  is,  in  general,  not 
available  or  not  credible,  and  because  some  tracks  contain  only  four  horizontal 
states,  it  is  recommended  that  track  and  truth  be  represented  as  state  vectors  in 
track-centered,  stabilized  east-north-up  (ENU)  Cartesian  reference  frames  (a 
different  frame  for  each  track).  For  track-to-truth  assignment,  use  only  the 
horizontal  position  and  velocity  states,  i.e., 

AXi,j  =  [(Pxi  -  Pxij),  (Py  -  Pyij),  (Vxi  -  Vxij),  (Vy  -  Vyij)]^ 

=  [(0  -  Pxij),  (0  -  Pyj),  (Vxi  -  Vxi,j),  (Vy  -  Vy,j)]T 
=  [(0  -  Pxij),  (0  -  Pyj),  (Si  sin  4)i  -  Vxij),  (Si  cos  4)i  -  Vyj)]^  (B-4) 


where  Pxij,  Pyj,  Vxij,  and  Vyj  are  the  truth  object  horizontal  position/velocity 
states  in  the  track  i-centered  ENU  frame.  Si  is  the  track  horizontal  speed,  and  (^i 
is  the  track  heading  measured  clockwise  from  north. 

For  some  local  track  databases,  sufficient  data  would  be  available  to 
completely  characterize  the  near-ideal  cost  function  in  this  four-dimensional 
simplification.  However,  this  is  not  generally  the  case  for  all  GTS  tracks.  Link  1 6 
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J3.2  air  track  reports,  for  instance,  do  not  provide  enough  data  to  decompose 
position  errors  into  orthogonal  directions.  For  common  treatment  of  remote  and 
local  tracks,  then,  it  is  recommended  to  model  track  error  as  independent  and 
equal  in  the  x  and  y  directions  of  the  track-centered  ENU  frame,  i.e.,  the  error 
covariance  is  modeled  as 


(P  +  Pj)  =  EPi,j 


^  Px,Vx 

0 


0 

0  <5vx 


*^Py,Vy 


0 


0 


^  Py.Vy 

0 


(B-5) 


where  o^px  =  o^py  =  o^p,  a^vx  =  =  o\,  and  opx,vx  =  opy,vy  =  op,v  and  are 

sums  of  the  corresponding  terms  in  the  Pi  and  Pj  error  covariance  matrices. 


Substituting  the  covariance  matrix  modeled  by  B-5  into  the  general 
unadjusted  cost  function  given  by  B-1  (without  the  gating  term),  the  unadjusted 
cost  becomes  the  following  two-dimensional  chi-square  distance  form: 


Q.(U)^ 


1 


{l(0-Px,)‘^+(0-Py)=]+p  [(S,sin(i-V)^)=+(S,co3|.,-Vyi)= 


V^Vy 


-2f 


(B-6) 


(0-Px)(^  sin^  -Vx)+(0-Py:)(SiCOS^^  -Vy)] } 


where  y  =  op,v/(ap  ov)  is  the  position-velocity  correlation  coefficient.  For  steady- 
state  air  vehicle  tracks,  y  and  (gf/ov)  are  approximately  constant  for  zero  predict 
ahead,  with  typical  values  of  y  =  0.65  and  (gr/gv)  =  4.3  to  9.0  seconds.  The 
experimental  basis  of  these  assertions  is  addressed  in  the  next  section  of  this 
appendix. 


Given  the  assumptions,  the  only  unknown  value  in  the  recommended 
unadjusted  cost  function  Ck(i,j)  is  gp.  Options  for  treatment  of  gp  include 
assuming  gp  is  constant  for  all  tracks  and  finding  the  best  gate  size  through 
experimentation.  The  advantage  of  this  approach  is  simplicity,  for  no  further 
assumptions  are  needed.  The  disadvantage  of  this  treatment  is  that  it  does  not 
use  all  available  information  to  achieve  the  best  track-to -truth  reconstruction. 


Another  option  is  to  estimate  gp  =  f[TQ(t)],  where  TO  =  track  quality.  For 
example,  compute  TQ(tu)  for  local  tracks  at  their  last  update  times  tu  before  the 
scoring  time  tk.  Use  the  TQ  value  in  remote  track  reports,  assuming  it  is  valid  at 
time  tr  of  receipt  of  the  report  just  before  tk.  Use  a  default  gate  size  of  28.5 
(about  0.99999  probability  gate),  or  find  the  best  gate  size  through 
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experimentation.  The  advantage  of  this  option  is  if  TQ  was  generated  in  a 
universal  way,  this  approach  makes  best  use  of  remote  track  data.  The 
disadvantage  of  this  approach  is,  in  the  near  term,  there  is  no  expectation  that 
TQ  will  be  calculated  in  a  universal  way. 

It  is  therefore  recommended  that  an  empirically  estimated  op  be  used  for 
each  track  source  in  the  scenario.  That  is,  pick  objects  for  which  the  assignment 
of  local  tracks  is  obvious  and  unambiguous,  calculate  the  kinematic  position 
accuracy  attribute  measure  with  the  vertical  state  weight  set  to  0,  divide  the  result 
by  the  square  root  of  two,  and  average  over  such  tracks.  Use  a  default  gate  size 
of  28.5  (about  0.99999  probability  gate)  or  find  the  best  gate  size  through 
experimentation.  The  advantage  of  this  option  is  that  it  provides  about  the  best 
that  can  be  done  in  the  absence  of  credible  covariance  and  universally  calculated 
TQ.  Unfortunately,  this  approach  also  requires  more  analysis  and  is  process 
intensive. 

The  adjusted  cost  function  presented  in  Section  3.3.2  of  the  main  text  is 
that  represented  by  equations  B-2,  B-3,  and  B-6  above,  along  with  the 
recommendation  just  described  for  empirical  estimation  of  op. 


Steady-State  Covariance  Values 

In  this  section,  background  details  and  experimental  data  are  provided  in 
support  of  the  claims  made  earlier  regarding  the  approximate  constancy  of 
certain  steady-state  covariance  parameters. 

The  Kalman  filter  algorithm  is  commonly  used  to  estimate  the  state  and 
error  covariance  of  a  system  from  the  measurements.  The  error  covariance  is  a 
measure  of  the  uncertainty  associated  with  the  state  estimate.  For  target 
tracking,  the  error  covariance  is  an  nx  n  symmetric  matrix,  where  n  is  the 
dimension  of  the  state  vector.  The  diagonal  elements  of  the  covariance  matrix 
are  the  mean  square  errors  associated  with  the  state  vectors.  The  off-diagonal 
terms  of  the  covariance  matrix  are  indicators  of  the  cross-correlation  between  the 
corresponding  errors  of  the  state  vector.  Under  certain  circumstances,  the  error 
covariance  can  reach  a  steady  state  and  become  time-invariant.  The  steady- 
state  error  covariance  is  assumed  in  computing  the  assignment  costs  in  the 
tracks-to -truth  assignment  step  in  computing  SIAP  attribute  measures. 

The  error  covariance  for  a  conventional  Kalman  filter  can  be  computed 
independent  of  the  measurements,  provided  that  measurement  errors  are  known 
at  each  time  and  the  state  and  measurement  equations  are  implemented  in  the 
same  coordinate  frame.  The  time  interval  between  updates  and  the  errors  for  the 
state  and  measurement  models  must  all  remain  constant  in  that  coordinate  frame 
to  obtain  a  steady-state  error  covariance  for  a  given  set  of  track  filter  parameters. 
Not  all  of  these  assumptions  are  realized  in  air  vehicle  tracking  situations,  but 
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making  these  assumptions  permits  the  derivation  of  values  needed  in  the  tracks- 
to-truth  assignment  cost  function. 

A  spherical  shaped  measurement  error  volume  was  employed  to  simplify 
derivation  of  the  values  need  in  the  tracks-to -truth  assignment  cost  function  (this 
maximizes  the  amount  of  uncertainty  applied  to  each  coordinate  state).  Under 
this  assumption,  the  state  and  measurement  models  are  decoupled  between 
each  of  the  coordinates,  and  the  measurement  error  is  also  independent  of 
sensor  position.  The  tracks-to -truth  assignment  cost  function  needs  covariance 
values  for  position  and  velocity  states  of  each  coordinate,  so  a  state  model  with 
constant  velocity  dynamics  is  employed.  The  Kalman  filter  with  a  constant 
velocity  motion  model  that  attains  steady-state  conditions  is  often  referred  to  as 
an  alpha-beta  filter  (Kalata,  1 984).  The  alpha  and  beta  values  can  be  used  to 
calculate  the  Kalman  gain  for  the  track  filter.  The  closed-form  solution  is 
determined  by  computing  the  error  covariance  at  the  current  time  using  the 
Kalman  filtering  equations,  equating  the  current  error  covariance  with  the 
previous  error  covariance,  and  then  solving  for  the  values  of  alpha,  beta,  and  the 
elements  of  the  covariance  matrix. 

Each  set  of  tracking  parameters  will  determine  unique  values  of  alpha  and 
beta.  The  tracking  parameters  needed  to  compute  the  error  covariance  are  the 
measurement  time  interval  and  the  errors  for  the  state  and  measurement  models. 
The  measurement  time  interval  and  the  measurement  model  error  are  functions 
of  the  sensor  and  usually  cannot  be  adjusted  by  the  tracking  algorithm.  The 
state  model  error  accounts  for  the  expected  amount  of  mismatch  that  exists 
between  the  state  model  and  the  true  model.  The  state  model  error  is  a  design 
parameter  which  is  usually  chosen  for  a  balance  between  the  needs  of  providing 
very  accurate  state  estimates  when  the  target  is  not  maneuvering  and  minimizing 
the  amount  of  error  during  maneuvering  periods. 

The  steady  state  error  covariance  values  for  a  1  -sec  measurement  interval 
(T,^),  a  measurement  error  (a,^)  of  250  meters,  and  a  state  model  error  (a„)  of 
25  m/s^'^  are  presented  in  Figure  B-1 .  Notice  the  covariance  values  obtain  a 
steady-state  condition  within  just  a  few  of  updates  after  initialization.  The 
covariance  value  for  position  is  indicated  by  Op,  the  value  for  velocity  is  indicated 
by  ,  and  the  value  for  the  cross-correlation  value  between  position  and  velocity 
is  indicated  by  Op^ .  The  tracks-to -truth  assignment  cost  function  needs  two 
quantities  to  be  computed  from  the  covariance  values.  One  is  the  ratio  between 
the  position  and  velocity  standard  deviations  and  is  computed  as  Op  /o^ .  The 
second  is  the  position-velocity  correlation  coefficient  (7 )  and  it  is  computed  as 
Op„  /OpO,, .  The  values  of  Op  and  7  computed  from  the  covariance  values  of 
Figure  B-1  are  presented  in  Figure  B-2.  These  typical  values  of  Op/o^  ~  4.3  sec 
and  7  =0.65  can  be  used  directly  in  the  tracks-to -truth  assignment  cost  function, 
provided  (1 )  the  method  of  time  aligning  for  computation  of  the  cost  function  is  to 
interpolate  the  truth  states  back  to  the  time  of  the  last  update  of  the  track  and  (2) 
the  track  is  a  local  track  (so  the  update  time  is  the  track  filter  update  time)  or  the 
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track  a  remote-only  track  in  which  the  time  stamp  (often  a  receipt  time  stamp)  is 
very  close  to  time  in  which  the  track  filter  on  the  remote  platform  was  updated 
with  a  sensor  measurement. 

If  the  track  states  are  predicted  for  significant  time  (e.g.,  remote-only  track 
states  were  predicted  due  to  delay  in  being  transmitted,  or  the  alternative  method 
of  assignment  time  aligning  is  employed  in  which  tracks  are  predicted  to  the 
scoring  time),  then  data  for  the  tracks-to -truth  assignment  cost  function  need  to 
be  predicted  to  the  desired  time  and  the  values  of  Op  /o^  and  y  computed  for  the 
assignment  cost  function.  Figure  B-3  presents  these  values  computed  as  a 
function  of  the  prediction  time  interval.  Notice  that  the  value  of  Op  /o^  increases 
in  a  near  linear  manner  while  the  value  of  y  remains  essentially  constant  for  this 
set  of  tracking  parameters.  The  steady-state  error  covariance  elements  and  the 
computed  and  predicted  values  of  Op  and  y  are  presented  in  Figures  B-4,  B- 
5,  and  B-6  for  T^=  10  sec,  a^=  500  meters,  and  =  15  m/s^'^.  This  set  of 
results  exhibits  the  same  trend  as  the  first  set  but  the  corresponding  value  of 
Gp  /g^  for  this  set  are  larger  due  to  the  significantly  increased  measurement  error 
and  sample  interval. 
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Figure  B-1 .  Covariance  values  for  parameter  set  1 . 
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T  =  1  sfto.  fi  =■  500 


=  25  m/s' 


Figure  B-2.  Values  of  Op  /o^  and  y  for  parameter  set  1 . 


T  =  1  s*o.  =  500  iT)«t«:rs.  =  25  m/s'  ® 


Figure  B-3. 


Predicted  values  of  Op  /o^  and  y  for  parameter  set  1 . 
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T  =  10  st*c  ci^  =  000  =  15  rrVft^ 


Figure  B-4.  Covariance  values  for  parameter  set  2. 


T  =  10  MC  ci^  =  500  =  15  nV*’  * 


Figure  B-5.  Values  of  Op  /a^  and  y  for  parameter  set  2. 


T  i  10  i«c  =  SOO  <»_  =  10  m/a*  ® 


Figure  B-6.  Predicted  values  of  Op  /a^  and  y  for  parameter  set  2. 
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APPENDIX  C 


Interpolation  of  Air  Vehicie  Truth  Trajectory  Data 

Air  vehicle  truth  trajectory  data  used  for  SIAP  assessments  is  expected  to 
be  scripted  or  recorded  at  discrete  times.  To  compute  assignment  costs  for 
tracks-to -truth  assignment  and  subsequently  to  score  tracks  for  kinematic 
accuracy,  as  will  as  to  compute  several  measures  of  performance,  the  air  vehicle 
truth  trajectories  must  be  interpolated  to  particular  times,  such  as  track  update 
times  or  scoring  times,  that  fall  between  the  discrete  times  at  which  the  truth 
trajectory  data  are  recorded.  This  appendix  outlines  practical  recommended 
methods  that  have  been  used  successfully  for  interpolation  of  truth  trajectory 
data. 


The  problems  of  interpolating  truth  trajectory  data  are  substantially 
different  for  environments  such  as  simulations  in  which  the  truth  state  data  are 
scripted  versus  empirical  situations  such  as  open-air  exercises  in  which  the  truth 
data  are  collected  from  instrumentation  or  navigation  subsystems  on  board  the 
air  vehicles  or  by  designated  sensors  such  as  test  range  radars.  Scripted  truth 
data  have  no  errors,  except  for  round-off  error  from  truncated  precision,  and  the 
scripted  data  can  include  position,  velocity,  acceleration,  and  any  other  states 
that  might  be  of  interest  for  tracks-to-truth  assignment  or  scoring.  In  most 
circumstances,  all  that  are  needed  to  interpolate  between  scripted  truth  data  are 
polynomial  fits  to  the  truth  data.  In  contract,  empirical  data  always  contains 
measurement  or  estimation  errors,  may  only  include  position  data,  and 
sometimes  include  spurious  data.  For  empirical  data,  some  form  of  filtering  and 
smoothing  is  usually  warranted  to  reduce  the  position  errors,  estimate  the 
velocity  states,  and  gate  out  spurious  data,  and  the  solutions  from  these  filtering 
and  smoothing  processes  form  the  best  basis  for  state  interpolation.  This 
appendix  treats  these  two  types  of  interpolation  separately. 


Interpolation  of  Scripted  Truth  Trajectory  Data 

Scripted  air  vehicle  truth  trajectory  data  are  typically  recorded  at  1  to  1 0 
Hz  data  rate  (1 .0  to  0.1  second  intervals)  and  include  a  time  and  the  air  vehicle 
position,  velocity,  and  acceleration  in  three  orthogonal  coordinates.  (It  may  also 
include  body  attitude  unit  vectors  and  even  body  attitude  angle  rate  data,  but 
these  are  not  of  interest  in  this  annex.)  The  formulas  in  this  annex  assume  the 
truth  trajectory  state  data  are  provided  in,  or  have  been  converted  to,  a 
convenient  Cartesian  reference  frame,  such  as  the  earth-centered,  earth-fixed 
(ECEF)  coordinates  defined  in  the  World  Geodetic  Standard  1984  (WGS  84)  or 
an  east-north-up  (ENU)  Cartesian  frame  centered  on  a  fix  latitude,  longitude, 
zero  altitude  location  on  the  WGS  84  ellipsoid.  SIAP  assessments  with  scripted 
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truth  data  need  air  vehicle  truth  states  at  any  time  during  the  duration  of  a 
scenario,  so  interpolation  of  truth  state  data  must  be  performed. 


The  interpolation  approach  in  this  section  is  based  on  using  only  the  truth 
state  data  recorded  at  the  bounding  scripted  times,  referred  to  here  as  Tast  and 
Tnext-  That  is,  for  interpolation  in  an  interval  between  Tast  and  Tnext,  the 
interpolation  scheme  will  not  use  truth  state  data  recorded  at  times  before  Tast  or 
after  Tnext-  This  avoids  complexity  of  having  to  invoke  different  interpolation 
methods  at  the  beginning  and  end  of  scenario  times,  when  for  one  side  there  is 
no  additional  state  data  recorded.  The  interpolation  formulas  also  meet  the 
requirement  that,  in  the  limit  as  time  approaches  one  of  the  bounding  times  with 
recorded  data,  the  interpolation  scheme  returns  the  truth  state  values  exactly  as 
recorded  at  the  bounding  time.  This  ensures  that  states  will  be  continuous 
across  the  entire  scenario  time.  Finally,  the  interpolation  formulas  minimize 
errors  due  to  both  kinematic  model  mismatch  and  recorded  data  precision,  while 
being  computationally  practical.  This  is  achieved  by  assuming  a  constant  jerk 
rate  kinematic  model  (i.e.,  a  fouth  order  polynomial)  during  the  interval,  but 
making  full  use  of  position,  velocity,  and  acceleration  state  data  recorded  at  the 
bounding  times  (i.e.,  six  data  values).  Since  only  five  coefficients  must  be  solved 
for  a  fourth  order  polynomial  and  six  data  values  are  available,  smoothing  is 
performed  to  mitigate  round-off  error  in  the  data  values. 

Assuming  a  constant  jerk  rate  kinematic  model  is  justified  from  both 
theoretical  and  experimental  perspectives.  Constant  jerk  (i.e.,  constant 
acceleration  rate)  during  an  interval  is  compatible  with  linear  change  of  air 
vehicle  body  axes  during  that  interval,  i.e.,  with  linear  change  in  air  vehicle  body 
attitude,  there  will  be  linear  change  in  the  angle  of  attack  of  aerodynamic  lift 
surfaces  and  a  corresponding  approximately  linear  change  in  the  rate  of  lift  force, 
giving  rise  to  a  linear  change  in  acceleration.  If  the  time  interval  between 
recorded  truth  data  is  small  enough,  changes  in  body  axes  can  indeed  appear  to 
be  linear  within  the  interval,  indicating  that  a  constant  jerk  (i.e.,  third  order 
polynomial)  kinematic  model  would  be  adequate  over  that  interval.  However,  as 
the  time  interval  grows  larger,  the  body  axis  change  tends  to  look  more  parabolic 
with  time  than  linear,  meaning  that  a  constant  jerk  rate  (i.e.,  fourth  order 
polynomial)  kinematic  model  is  more  appropriate.  In  experimental  tests  of  the 
ability  to  reproduce  closely  the  position  and  velocity  states  from  a  full  6-degree- 
of-freedom  aerodynamic  object  model  by  interpolating  with  position,  velocity,  and 
acceleration  data,  it  was  found  that  a  constant  jerk  rate  kinematic  model  did 
support  significantly  larger  time  intervals  before  accuracy  tolerances  were 
exceeded.  This  could  be  taken  a  step  further  to  argue  that  a  fifth  order 
polynomial  kinematic  model  would  permit  even  larger  time  intervals.  However, 
solving  for  the  coefficients  of  a  fifth  order  polynomial  requires  all  the  position, 
velocity,  and  acceleration  data  values  at  the  bounding  times,  leaving  no  option 
for  smoothing  to  mitigate  round-off  error,  as  is  possible  using  a  fourth  order 
polynomial.  Tolerance  against  round-off  error  is  considered  a  useful 
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characteristic  of  the  interpolation  scheme,  so  a  constant  jerk  rate  (fourth  order 
polynomial)  kinematic  model  is  a  good  compromise. 

Position  Interpoiation  from  Scripted  Truth  Data 

Since  constant  jerk  rate  is  assumed  during  the  interval,  the  equation  for 
position  is  a  fourth  order  polynomial  in  time,  having  five  parameters  (coefficients). 
However,  there  are  a  total  of  six  truth  states  available  from  the  boundary 
conditions:  position,  velocity,  and  acceleration  atTiast  and  position,  velocity,  and 
acceleration  at  Tnext-  Thus,  many  different  constant-jerk  rate  solutions  are 
feasible  and  they  can  be  blended  to  mitigate  substantially  the  effects  of  round-off 
error  in  the  recorded  truth  data  at  the  boundaries.  The  criteria  employed  for 
selecting  the  position  interpolation  scheme  that  is  presented  here  were  to  (1) 
minimize  the  worst-case  error  and  then  (2)  minimize  the  square  root  of  the 
average  error  variance  over  the  entire  interval.  After  examining  many  different 
combinations,  it  was  found  that  the  weighted  sum  of  a  solution  formed  by  using 
the  boundary  conditions  {Piast,  Viast,  Aiast,  Pnext,  Vnext}  and  one  formed  by  using  the 
boundary  conditions  {Piast,  Viast,  Pnext,  Vnext,  Anext},  referred  to  as  the  left-side  and 
right-side  solutions,  respectively,  had  better  (possibly  the  best  feasible)  potential 
for  minimizing  errors.  This  is  the  scheme  described  below. 

For  symmetry,  define  Trei  as  the  time  relative  to  the  midpoint  time  of  the 
interval. 


T.e,=T 


“^next  Tast 


(C-1) 


and,  for  normalizing,  define  AT  as  duration  from  the  midpoint  of  the  interval  to 
either  boundary. 


AT  = 


Tast  Xiexl  _  -p 


last 


(C-2) 


The  left-side  interpolation  formula  for  position,  i.e.,  the  one  from  the  boundary 
conditions  (Piast,  Viast,  Aiast,  Pnext,  Vnext},  is  given  by 
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-2^ P,..,  -3P„.d  +6y„AT  +2A..,Ar  j  i 


+4[F1„,  -  P„«  +  y„,AT  +  V„„,AT]  J 


A 


(C-3) 


f  Y 

+  [sF;..,  -  3P„  +4V,.„AT  +  2V„„,AT  +  2A,.„AT^1  ^  j  } 

The  right-side  interpolation  formula  for  position,  i.e.,  the  one  from  the  boundary 
conditions  {Piast,  Viast,  Pnext,  Vnext,  Aiast},  is  given  by 

^right  -side  ( Xel )  “  ^0  ^  ^next  ^  ^  Y  ast  AT  -  8  T  +2  A^g^^j  AT  ] 


-  4[3F>3st  -  3P.ex,  +  MastAT  +  Vgg^YTj  ^ 


21  Xg| 

AT 


A 


+  2  f  P,..,  -3P„.„  +  6  V„.,  AT  -  2  A„„  AT 
+ -  p„„,  +  y,„AT  +  v„„  at!  i 

f  Y 

-^Pi.,,  -3P„,  +  2y,.,AT  +  4V„^T-2A„,„AT^|i  I  } 


(C-4) 


The  weighted  blending  of  the  left-  and  right-side  solutions  is  then  just 

P(Xe,)  =  ^pos  (Trei  )P|eft-side  (Xel )  +  P  ^pos  (TJ] 

Pright  -side  (Xei)  (C-5) 

for  a  weighting  function  Wpos(Trei).  It  can  be  shown  that  the  simple  weight 
Wpos(Trei)  =  1/2  will  give  the  smallest  maximum  error,  but  is  not  optimum  with 
respect  to  minimizing  the  average  variance  over  the  entire  time  interval.  The 
average  variance  can  be  improved  by  specially  chosen  higher  order  polynomial 
functions  of  Trei,  but  these  result  in  larger  maximum  error  magnitude.  Given  this 
dilema  and  the  general  desire  to  keep  the  interpolation  formula  simple,  a  weight 
of  Wpos(Trei)  =  1/2  is  recommended. 
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Velocity  Interpolation  from  Scripted  Truth  Data 


Since  constant  jerk  rate  is  assumed  during  the  interval,  the  equation  for 
velocity  is  a  third  order  polynomial  in  time,  having  four  parameters  (coefficients). 
In  theory,  there  are  a  total  of  six  truth  states  available  from  the  boundary 
conditions;  however,  use  of  position  states  in  the  velocity  interpolation  formulas 
introduces  an  extra  AT  in  the  denominators  of  terms  containing  Piast  or  Pnext, 
thereby  amplifying  the  effects  of  their  round-off  errors.  Consequently,  unless 
position  states  were  to  be  recorded  with  higher  precision  in  the  truth  files,  the 
useful  set  of  boundary  conditions  for  velocity  interpolation  is  just  four  states: 
velocity  and  acceleration  at  Tiast  and  velocity  and  acceleration  at  Tnext-  Since  all 
four  boundary  condition  are  needed  as  polynomial  coefficients,  only  one  feasible 
solution  remains  {Viast,  Aiast,  Vnext,  Anext}- 


Using  the  same  Trei  and  AT  definitions  as  for  position  interpolation,  the 
interpolation  formula  for  velocity  is  given  by 


v(Xj 


^2^ 

v4y 


{  [2V|ast  +  2yiext  +  AasfAT  -  A^gxfAT ] 


[3  V,„,  -  3V„,  +  A,„,AT  +  A„„,AT]  i 


■  [^last 


AT-A„,AT|i 
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AT 


A3 


+  [yast-Vnext+A,33,AT+A„,,.AT]^^J  ] 


(C-7) 


Interpolation  of  Empirical  Truth  Trajectory  Data 

The  determination  of  the  interpolated  ground  truth  position  and  velocity 
states  for  the  tracks-to -truth  assignment  and  for  performance  scoring  is  much 
more  difficult  for  empirical  data  as  compared  to  interpolation  of  scripted  truth 
data.  Interpolation  using  higher  order  state  derivatives  may  not  be  applicable 
when  using  empirical  data,  since  the  data  employed  in  the  truth  determination 
may  have  large  errors  that  make  the  use  of  such  an  interpolation  method 
unreliable.  Moreover,  the  empirical  data  may  only  provide  position  states. 
Conventional  filtering  and  least-squares  techniques  can  be  employed  to  estimate 
the  position  and  velocity  states  at  arbitrary  times,  but  these  techniques  fall  short  if 
very  accurate  trajectory  state  reconstruction  is  needed.  For  these  reasons, 
smoothing  is  recommended  for  estimating  (interpolating)  ground  truth  position 
and  velocity  from  empirical  data. 
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Estimates  of  the  air  vehicle  true  position  and  velocity  states  can  be 
obtained  by  sequentially  filtering  the  position  measurements  and  calling  the 
outputs  the  ground  truth.  This  can  be  accomplished  using  a  single  kinematic 
model  or  more  accurately  using  a  multiple  model  estimation  technique,  such  as 
the  Interacting  Multiple  Model  (IMM)  algorithm  (Bar-Shalom,  Gang,  and  Blom, 

1 989;  Blair  and  Watson,  1 997;  Blom  and  Bar-Shalom,  1 988;  Tugnial,  1 982).  The 
time  at  which  the  state  must  be  known  (the  interpolation  time)  will  not,  in  general, 
coincide  with  a  measurement  time.  Thus,  prediction  is  required,  which  will 
introduce  even  more  errors  into  the  state  estimate  of  ground  truth. 

Fixed-Interval  Smoothing 

Fixed-Interval  Smoothing  (FIS)  is  a  more  reliable  reconstruction  technique 
when  compared  to  conventional  forward-time  filtering  because  it  uses  the  past, 
present,  and  future  position  measurements  from  the  data  set  to  estimate  the 
target  state  (Blair  and  Watson,  1997;  Flelmick,  et.  al.,  1993, 1995,  and  1996). 
Single  model  and  multiple  model  FIS  algorithms  have  been  employed 
successfully  to  reconstruct  the  ground  truth  trajectories  for  live  experiments  (Blair 
and  Watson,  1 997,  and  Watson,  2001 ).  The  FIS  is  accomplished  as  follows. 

First,  an  estimate  of  the  state  is  computed  in  the  forward-time  direction  with  all 
the  measurements.  Next,  an  estimate  of  the  state  is  computed  in  the  backward¬ 
time  direction  with  all  the  data.  The  last  step  is  to  combine  optimally  the  forward¬ 
time  and  backward-time  estimates  to  achieve  a  more  accurate  state  estimate 
than  the  individual  directions  alone.  The  estimation  error  at  the  onset  and 
completion  of  maneuvers  should  be  much  smaller  or  vastly  improved  using  this 
method  by  reducing  the  error  associated  with  the  lag  of  each  filter. 

To  accomplish  the  smoothing  process,  the  forward-time  and  backward¬ 
time  Kalman  filter  estimates  and  associated  error  covariances  at  each  position 
measurement  time  should  be  stored.  (For  a  multiple  model  filter,  the  state 
estimates,  error  covariances,  and  mode  probabilities  for  each  model  must  be 
stored.  There  is  a  significant  increase  in  required  storage  when  using  a  multiple 
model  approach.)  The  use  of  forward  and  backward  filtering  is  done  once  and 
the  stored  filter  estimates  and  error  covariance  are  then  used  for  all  subsequent 
ground  truth  state  interpolation. 

The  smoothed  estimate  must  be  computed  at  a  particular  interpolation 
time.  This  requires  that  the  forward-time  and  backward-time  estimates  be 
predicted  and  properly  fused  using  a  minimum  variance  method.  The  smoothed 
estimate  at  the  interpolation  time  will  be  more  accurate  than  the  conventional 
filtered  states  predicted  to  that  time,  since  future  data  are  employed  in  the 
smoothing  process.  The  smoothed  state  is  accomplished  according  to  the 
following  three  steps: 

Time  alignment  of  the  filtered  data; 

Fusing  of  the  state  estimates; 

Smoothing  measurement  update. 
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Step  1 :  Time  Alignment  of  Filtered  Data 


For  interpolation  time  tk,  the  forward-time  data  employed  in  the  smoothing 
process  are  selected  to  be  the  Kalman  filter  estimate  that  is  closest  to  but  not 
exceeding  tk.  The  backward-time  data  employed  in  the  smoothing  process  is 
selected  to  be  the  Kalman  filter  estimate  that  is  closest  to  but  not  preceding  tk. 
The  state  prediction  process  of  the  Kalman  filter  is  then  applied  to  the  forward¬ 
time  and  backward-time  filtered  estimates  to  make  them  coincident  with  tk.  The 
following  data  must  be  computed  from  the  stored  filtered  data  for  a  single  model 
track  filter: 


X[|k  =  forward  -  time  state  estimate  predicted  to  t^ 

=  backward  -  time  state  estimate  predicted  to  t^ 
Pk|k^  =  forward  -  time  error  covariance  predicted  to  t^ 

=  backward  -  time  error  covariance  predicted  to  t^ 


with  kf  and  kb  being  the  times  of  the  most  recent  forward-time  and  backward-time 
estimates,  respectively. 

Step  2:  Fusing  of  State  Estimates 

The  forward-time  and  backward-time  state  estimates  and  error 
covariances  are  fused  using  a  minimum  variance  approach  given  by 

=  p„  ((PkV,  r  x:,,  +  (Pi, )-'  ^ 

Pkin = ((pkV,  r + (pi.  r  r 

Step  3:  Smoothing  Measurement  Update 

The  smoothed  state  estimate  and  error  covariance  at  tk,  X^^  and  Pk®^ ,  are 

computed  with  the  fused  values  from  Step  2  and  the  measurement  at  tk  if  one 
exists.  There  are  two  cases  to  consider  since  a  measurement  may  not  be 
available  at  tk. 

Case  1 :  If  there  is  a  measurement  at  tk,  the  measurement  update  is 
accomplished  with  the  Kalman  filtering  equations  given  by 


Y®  —  Y  4-  k"  7 

^k|N  ~  ■''■kIN  ''‘k|N  ^k|N 

^k|N  ~  [l  ~^k|M  ^k]^k|N 


(C-9a) 
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where 


with 


ys  _  7  _  |_|  y 

^k|N  “  "^k  ^'klN 

^k|N  “  f^k|N  kik  Sk|N 
^k|N  “  kl  k  Pk|N  H  k  +  R  k 


(C-9b) 


Zk  =  measurement  at  tk 

Rk  =  covariance  of  Zk 

Hk  =  gradient  of  Zk  with  respect  to  Xk 

Case  2:  If  there  is  no  measurement  at  tk,  the  smoothed  state  estimate  and  error 
covariance  are  given  by 


vs  _  Y 

^k|N  “  -^klN 

P®  =P 

•^klN  '^k|N 


(C-10) 


Simulation  results  comparing  the  FIS  with  the  forward-time  and  backward-time 
filters  are  presented  for  a  single  constant  velocity  model.  The  simulated  target 
shown  in  Figure  C-1  is  employed  for  this  example.  The  target  has  an  initial  range 
of  83  km,  speed  of  457  m/s,  and  altitude  of  3.0  km.  The  air  vehicle  flies  straight 
and  level  for  the  first  30  seconds.  A  4-g  turn  is  then  performed  through  a  45 
degree  course  change.  Straight  and  level,  non-accelerating  flight  is  continued  for 
the  next  30  seconds.  A  second  4-g  turn  through  a  90  degree  course  change  is 
performed  while  the  aircraft  decelerates  to  a  speed  of  274  m/s.  Straight  and 
level  flight  is  maintained  for  the  remainder  of  the  flight  after  the  course  change  is 
completed.  The  root-square-error  (RSE)  of  position  and  velocity  computed 
between  the  estimates  and  true  states  at  each  measurement  time  is  presented  in 
Figure  C-2.  The  error  of  the  smoother  is  significantly  less  than  the  forward 
filtering  or  the  backward  filtering  over  the  entire  length  of  the  track  and  it  is 
especially  noticeable  during  target  maneuvers.  The  uncertainty  in  the  estimates, 
represented  by  the  Filter  Generated  Standard  Deviation  (FGSD),  is  also  much 
smaller  for  the  smoother.  The  FGSD  is  a  measure  of  how  well  the  filter 
perceives  its  own  performance.  The  single  model  FIS  employed  in  the  example 
illustrates  the  improved  performance  that  can  be  obtained  for  trajectory 
reconstruction  using  smoothing  when  compared  to  a  conventional  filtering 
approach.  Even  though  the  error  covariance  is  small,  the  single  model  FIS  has 
the  same  drawback  as  the  conventional  Kalman  filter  by  not  responding  to  the 
target  maneuvers.  The  single  model  filter  does  not  provide  consistent  estimates 
since  its  error  covariance  does  not  increase  to  reflect  the  degradation  in 
performance  during  maneuvers.  A  multiple  model  FIS  can  provide  an  error 
covariance  that  is  nearly  consistent  while  at  the  same  time  providing  more 
accurate  estimates  (Blair  and  Watson,  1 997). 
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For  implementation,  the  forward-time  and  backward-time  estimates  will 
need  to  be  computed  and  stored  prior  to  the  evaluation  process  in  order  execute 
in  a  timely  manner.  However,  the  accuracy  of  the  estimates  may  be  so  good  that 
an  interpolation  scheme  could  be  employed  over  portions  of  the  reconstructed 
trajectory  using  the  smoothed  estimates  obtained  for  each  measurement  time. 
The  use  of  an  interpolation  scheme  would  reduce  the  amount  of  stored  data 
needed  to  compute  the  trajectory  state  at  the  interpolation  time. 
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Figure  C-1 .  Target  profile. 
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Figure  C-2.  Performance  comparison. 
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Figure  C-2.  Performance  Comparison,  (continued) 
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